Some images are created using different filesystems, most popular
squashfs and ext4. To allow downstream projects to distinguesh between
those, add the `filesystem` information to created json files.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The previous behavior prefered same feed for dependent packages as
initial package. This caused inconsitency in installation of packages.
The difference was if two feeds provide same package (different version)
there was different result if you executed install for that specific
version compared to install for package depending on it from different
feed.
This ensures that preferred feed is propagated without change and
selected feed is used only really for package it was selected for.
Signed-off-by: Karel Kočí <karel.koci@nic.cz>
Otherwise, a n00b like myself can get quite confused when moving a
package from core to feeds, for example.
(Hint: one *really* needs to clear out the tmp/info/.packageinfo...
entries for the stale package, but '-f' works as well.)
Signed-off-by: Brian Norris <computersforpeace@gmail.com>
It seems like after a build the /dl dir seems to now contain a .hash
file for each source file due to inproper cleanup so fix it by removing
those intermediate files before leaving the download action.
Fixes: 4e19cbc553 ("download: handle possibly invalid local tarballs")
Reported-by: Hannu Nyman <hannu.nyman@iki.fi>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Currently it's assumed, that already downloaded tarballs are always
fine, so no checksum checking is performed and the tarball is used even
if it might be corrupted.
From now on, we're going to always check the downloaded tarballs before
considering them valid.
Steps to reproduce:
1. Remove cached tarball
rm dl/libubox-2020-08-06-9e52171d.tar.xz
2. Download valid tarball again
make package/libubox/download
3. Invalidate the tarball
sed -i 's/PKG_MIRROR_HASH:=../PKG_MIRROR_HASH:=ff/' package/libs/libubox/Makefile
4. Now compile with corrupt tarball source
make package/libubox/{clean,compile}
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Silence the warning in git 2.27 about undefined fast-forward style
in git pull. Define "ff-only" as the style.
Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
As package size changes are a continuous topic on the mailing list this
scripts helps developers to compare their local package modifications
against latest upstream.
The script downloads the latest package indexes based on env variables
or the `.config` file. The script compares the actual installed size
(data.tar.gz) or the IPK package size.
An example output is found below:
```
user@dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh
Compare packages of ath79/tiny/mips_24kc:
dropbear busybox iw ubus
Checking configuration difference
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 554 100 554 0 0 336 0 0:00:01 0:00:01 --:--:-- 336
--- start config diff ---
--- /tmp/config.DDjwVh-LOCAL 2020-11-23 09:08:28.913203068 -1000
+++ /tmp/config.DDjwVh-UPSTREAM 2020-11-23 09:08:36.369240887 -1000
@@ -1,5 +1,9 @@
+CONFIG_ALL_KMODS=y
+CONFIG_ALL_NONSHARED=y
CONFIG_AUTOREBUILD=y
+CONFIG_AUTOREMOVE=y
--- 8< ---
CONFIG_BINARY_FOLDER=""
+CONFIG_BUILDBOT=y
+CONFIG_TARGET_ALL_PROFILES=y
CONFIG_TARGET_ROOTFS_DIR=""
CONFIG_USE_SSTRIP=y
CONFIG_USE_UCLIBCXX=y
--- end config diff ---
Checking installed size
Fetching latest package indexes...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 80634 100 80634 0 0 33499 0 0:00:02 0:00:02 --:--:-- 33485
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 54082 100 54082 0 0 24252 0 0:00:02 0:00:02 --:--:-- 24252
Comparing package sizes...
Change Local Remote Package
+271 51386 51115 base-files
+123 705241 705118 bnx2-firmware
+86 17209 17123 fstools
+22 47989 47967 procd
+21 208311 208290 busybox
+19 67181 67162 netifd
```
I plan to integrate this script into the CI so we have a summary how
sizes change over different architectures.
Signed-off-by: Paul Spooren <mail@aparcar.org>
With this commit, the download script will try downloading source files
using the filename instead of the url-filename in case the previous
download attempt using the url-filename failed.
This is required, as the OpenWrt sources mirrors serve files using the
filename files might be renamed to after downloading. If the original
mirror for a file where url-filename and filename do not match goes
down, the download failed prior to this patch.
Further improvement can be done by performing this only for the
OpenWrt sources mirrors.
Signed-off-by: David Bauer <mail@david-bauer.net>
Kernel has separated the structs that are reported to be const in
checkpatch.pl into a file of its own, const_structs.checkpatch.
This file has been missing after the recent update of checkpatch.pl,
leading to the following message:
No structs that should be const will be found - file
'/data/openwrt/scripts/const_structs.checkpatch': No such file
or directory
This commit adds the relevant file from v5.10-rc4.
Fixes: 086ee09bbc ("scripts: Update checkpatch.pl to 2020-06-11")
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
The kernel expects changes to MAINTAINERS for all removed or added
files, printing warnings like:
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#828:
deleted file mode 100644
Since this does not apply to "our" files in OpenWrt repo, this
warning should be disabled.
This can be achieved easiest by setting $reported_maintainer_file
to 1. While this is a hack that tricks the script into believing
the proper MAINTAINERS changes have been made, it's the easiest
solution as it does not require to touch any other code.
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
The source date epoch is the only reproducible date close to the actual
build date. It can be used for tooling like the firmware wizard to show
the image age.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The f-string feature was introduced in Python 3.6. As Buildbots may run
on Debian 9, which comes per default with Python 3.5, this would cause
an issue. Instead of f-strings use the *legacy* `.format()` function.
Signed-off-by: Paul Spooren <mail@aparcar.org>
On a system python3 is linked to python3.6, fail to perform json_overview_image_info
and got `TypeError: __init__() got an unexpected keyword argument 'capture_output'`.
This patch emulate the behaviour on python 3.7+.
Signed-off-by: Huangbin Zhan <zhanhb88@gmail.com>
Only collect arch_packages if actually generating any output.
Fixes: commit f09b9319 ("build,json: store arch_packages in profiles.json"(
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>
The `arch_packages` contains the supported package architecture.
Previously it was necessary to parse the `Packages` index for the line
`Architecture:`, requiring both an additional parser and file download.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The JSON `WORK_DIR` ($(KDIR)/json_info_files) is only created if the new
image generation methods from `image.mk` are used. However some targets
like `armvirt` do not use it yet, so the folder is never created.
The `json_overview_image_info.py` script used to raise an error if the
given `WORK_DIR` isn't a folder, however it should just notify about
missing JSON files.
This patch removes the Python assert and exists with code 0 even if no
JSON files were found, as this is not necessarily an error but simply
not yet implemented. Using `glob` on an not existing `Path` results in
an empty list, therefore the for loop won't run.
Signed-off-by: Paul Spooren <mail@aparcar.org>
CC: Petr Štetiar <ynezz@true.cz>
With this commit the `profiles.json` contain both the target specific
`default_packages` as well as the device specific `device_packages` as a
array of strings.
This information is required for downstream projects like the various
web-based interactive firmware generators.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>
JSON info files contain machine readable information of built profiles
and resulting images. These files were added in commit 881ed09ee6
("build: create JSON files containing image info").
They are useful for firmware wizards and script checking for
reproducibility.
Currently all JSON files are stored next to the built images, resulting
in up to 168 individual files for the ath79/generic target.
This patch refactors the JSON creation to store individual per image
(not per profile) files in $(BUILD_DIR)/json_info_files and create an
single overview file called `profiles.json` in the target directory.
Storing per image files and not per profile solves the problem of
parallel file writes. If a profiles sysupgrade and factory image are
finished at the same time both processes would write to the same JSON
file, resulting in randomly broken outputs.
Some target like x86/64 do not use the image code yet, resulting in
missing JSON files. If no JSON info files were created, no
`profiles.json` files is created as it would be empty anyway.
As before, this creation is enabled by default only if `BUILDBOT` is set.
Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[json_info_files dir handling in Make, if case refactoring]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Running the updated checkpatch version with spelling.txt show that
spelling mistakes happen to everyone.
> /target/linux$ fd .*.patch | xargs ../../scripts/checkpatch.pl | rg spell
WARNING: 'usefull' may be misspelled - perhaps 'useful'?
WARNING: 'afecting' may be misspelled - perhaps 'affecting'?
WARNING: 'usefull' may be misspelled - perhaps 'useful'?
WARNING: 'afecting' may be misspelled - perhaps 'affecting'?
WARNING: 'begining' may be misspelled - perhaps 'beginning'?
WARNING: 'superflous' may be misspelled - perhaps 'superfluous'?
WARNING: 'multipe' may be misspelled - perhaps 'multiple'?
WARNING: 'recieves' may be misspelled - perhaps 'receives'?
WARNING: 'retreive' may be misspelled - perhaps 'retrieve'?
WARNING: 'tranfer' may be misspelled - perhaps 'transfer'?
WARNING: 'additonal' may be misspelled - perhaps 'additional'?
WARNING: 'accomodate' may be misspelled - perhaps 'accommodate'?
[...]
Signed-off-by: Paul Spooren <mail@aparcar.org>
Remove checks for device tree documentation as the OpenWrt tree comes
withouth the ./Documentation folder.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Checkpatch grown in functionallity and we should make use of that. If
OpenWrt patches should be upstream material they should also be checked
based on upstream checkpatch.pl instead of 2013.
Signed-off-by: Paul Spooren <mail@aparcar.org>
mkhash currently returns the hash of an empty input when trying to hash
a folder. This can be missleading in caseswhere e.g. an env variable is
undefined which should contain a filename. `mkhash ./path/to/$FILE`
would exit with code 0 and return a legit looking checksum.
A better behaviour would be to fail with exit code 1, which imitates the
behaviour of `md5sum` and `sha256sum`.
To avoid hashing of folders the `stat()` is checked.
Hashing empty inputs result in the following checksums:
md5: d41d8cd98f00b204e9800998ecf8427e
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Signed-off-by: Paul Spooren <mail@aparcar.org>
If hashing a file fails mkhash shouldn't just silently fail. Now check
after each call of `hash_file()` the return and exit early in case of
errors. The return value which was previously ignored and would always
return 0.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The -n option prints the filename of hashed files next to the calculated
checksum. Reflect that in the usage message.
user@dawn:~/src/openwrt/openwrt$ ./a.out md5 -n .config
eb06db36e7b6751cb18801945e46bf5d .config
Signed-off-by: Paul Spooren <mail@aparcar.org>
Recent versions (> 5.33) of `file` report liblto_plugin.so as
executable:
$ file liblto_plugin.so
liblto_plugin.so.0.0.0: ELF 64-bit LSB pie executable ...
Which then leads to improper packaging of the plugin, resulting in the
broken compiler:
configure: checking whether the C compiler works
mips-openwrt-linux-musl/bin/ld: liblto_plugin.so: error loading plugin: liblto_plugin.so: invalid ELF header
As the LTO compiler plugin library is incorrectly packaged as SDK
executable:
$ head -1 ~/staging_dir/toolchain...libexec/gcc/.../liblto_plugin.so
#!/usr/bin/env bash
Fix this by filtering out shared libraries from the patching.
Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1296868
Acked-by: Jo-Philipp Wich <jo@mein.io>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Shallow copies are quite common on CI platforms nowadays, making REBOOT
tag unavailable, thus producing following confusing errors in the build
logs:
fatal: Invalid revision range ee53a240ac902dc83209008a2671e7fdcf55957a..HEAD
fatal: Invalid revision range ee53a240ac902dc83209008a2671e7fdcf55957a..0493d57e04774d47921a7d2014b567455d5dc16b
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Explicitly set the QEMU cpu type and support the MIPS R2 ISA, for both
64-bit and 32-bit targets. The later previously supported MIPS R2 by
implicit default.
This is needed after commit 93608697f3 ("malta: update MIPS64 ISA to R2"),
otherwise booting malta images with scripts/qemustart will hang.
Signed-off-by: Tony Ambardar <itugrok@yahoo.com>
SPDX moved from GPL-2.0 to GPL-2.0-only and from GPL-2.0+ to
GPL-2.0-or-later. Reflect that in the SPDX license headers.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[adjust commit title, update remaining files]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
As the PWM has its own sub-system in the Linux kernel,
I think it should be handled in the same way as GPIO, RTC, PCI...
This patch introduces a specific feature flag "pwm" and the
"leds-pwm" kernel module as the first customer.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Some bootloaders are really keen on just one special
fdt in a multi-image fit image. This is a problem, because
currently this is fixed to "fdt@1".
This patch introduces a new device variable:
DEVICE_FDT_NUM that allows to specify the right
fdt number.
If the value is absent "1" will be chosen.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Use the prepared .packageusergroup file to lookup user and group names
when processing the passed file mode.
Also replace the various subshell/cut invocations with a sequence of
standard variable interpolations which fixes paths with embedded colons
as a side-effect.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
With the introduction of `./tmp/userids` the `ipkg-build` script can now
resolve values of "PKG_FILE_MODES", allowing users to set names rather
than numeric values.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Since the introduction of fakeroot support, wrapped SDK executables might
be invoked from a shell that has libfakeroot.so preloaded.
Since we're using preloading as well in order to mangle argv[0] when
invoking the shipped ELF interpreter directly, we must take care of
preloading the already preloaded libraries as well, to avoid invoked
programs losing their fakeroot capabilities.
Extend the bundle-libraries.sh script to take any existing $LD_PRELOAD
into account when invoking the target ELF executable with a preloaded
runas.so library.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
With the new `SOURCE` argument of `get_source_date_epoch` it is possible
to set package timestamps based on actual package changes rather thane
$TOPDIR changes.
This commit adds a new variable PKG_SOURCE_DATE_EPOCH which is used by
the `ipkg` build script. As a fallback the existing SOURCE_DATE_EPOCH is
used or as last resort the current time.
The redundant checks for `.git/` and `.svn/` are removed.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The SOURCE_DATE_EPOCH variable is used to make builds reproducible even
if rebuild at different times. Instead of using the current timestamp,
the time of the last source change is used.
Created packages are `touch`ed with a specific timestamp so resulting
packages have the same checksums.
The `get_source_date_epoch.sh` script tries multiple ways (file, git,
hg) to determine the correct timestamp.
Until now the script would only consider the $TOPDIR instead of package
specific changes. Resulting in packages with same versions but different
timestamps, as $TOPDIR (openwrt.git) received changes not affecting
package versions. This results in warning/erros in `opkg` as the package
versions stay the same but checksums changed.
This commit adds an optional argument to get the `SOURCE_DATE_EPOCH` of
a specific path (e.g. package SOURCE) rather than the $TOPDIR. As a
consequence this allows granular but still reproducible timestamps.
As packages might be distributed over multiple repositories the check
for `.git/` becomes unfeasible. Instead tell `git` and `hg` to change
their working directories and automatically traverse the repo folder.
Signed-off-by: Paul Spooren <mail@aparcar.org>
