Commit Graph

252 Commits

Author SHA1 Message Date
AmadeusGhost
5680bfc347 Merge Mainline 2021-04-21 10:50:08 +08:00
Tianling Shen
1b76d7006b
scripts/download.pl: only call our own mirror when failed to download
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-04-19 01:36:37 +08:00
Tianling Shen
29b53e375a
scripts/download.pl: add aliyun(cn) mirror
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-04-19 00:15:44 +08:00
Tianling Shen
e46eaffee2
Merge Mainline
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-04-17 00:17:46 +08:00
Tianling Shen
9961791b9e
scripts/download.pl: remove unavailable mirror
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-04-17 00:12:02 +08:00
Tianling Shen
4f53a26f06
scripts/download.pl: update mirrors
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-04-13 21:33:25 +08:00
John Doe
4e84824d42
Merge Mainline
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-29 18:08:36 +08:00
Tianling Shen
3364f9a63d
scripts/download.pl: re-order mirrors
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-26 22:16:38 +08:00
Tianling Shen
8cb43623b1
scripts/download.pl: correct branch name of self-hosted mirror
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-26 21:33:50 +08:00
AmadeusGhost
b9ae8b7c93 Merge Mainline 2021-03-26 20:28:03 +08:00
Paul Spooren
5ec2e9b8c7 build,json: 3rd fixup of default_packages
This became a bit of a tragedy, caused by a corner cases which wasn't
put into account during testing. DEFAULT_PACKAGES are defined in
target/linux/<target>/Makefile but also in
target/linux/<target>/<subtarget>/target.mk.

The latter was no longer imported when using DUMP=1, however not using
DUMP=1 while running the Makefile in target/linux/<target>/ caused duplicate
packages in the list.

As a solution, which should have been used from day 0, `make` runs in
target/linux/ without DUMP=1, resulting in no duplicate packages and all
inclusions from include/target.mk, linux/target/<target>/{Makefile,
<subtarget>/target.mk}

While at it, sort the list of default packages.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-26 20:00:19 +08:00
Paul Spooren
d28cf096e3 build,json: fixup fixup of arch_packages
The commit "1bf2b3fe90 build,json: fixup missing arch_packages" fixes
the missing package architecture locally but runs $(TOPDIR)/Makefile
rather than a target specific one. While this works on local builds just
fine, it causes the buildbots to add garbage to the `arch_packages`
variable:

    cd \"/builder/shared-workdir/build\"; git log --format=%h -1
    toolchain > /builder/shared-workdir/build/tmp/.ver_check\ncmp -s
    /builder/shared-workdir/build/tmp/.ver_check
    /builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/stamp/.ver_check
    || { \\\n\trm -rf
    /builder/shared-workdir/build/build_dir/target-x86_64_musl
    /builder/shared-workdir/build/staging_dir/target-x86_64_musl
    /builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl
    /builder/shared-workdir/build/build_dir/toolchain-x86_64_gcc-8.4.0_musl;
    \\\n\tmkdir -p
    /builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/stamp;
    \\\n\tmv /builder/shared-workdir/build/tmp/.ver_check
    /builder/shared-workdir/build/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/stamp/.ver_check;
    \\\n}\nx86_64

Only the last line contains the desired string.

Future investigation should check why the build system prints this to
stdout rather than stderr.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-25 23:38:15 +08:00
Paul Spooren
65ca8a8b72 build,json: fixup missing arch_packages
Fix 7f4c2b1 "build,json: fix duplicates in default_packages" which
removed duplicate default packages but also removed the package
architecture from the profiles.json.

If DUMP=1 is set, the `ARCH_PACKAGES` is no longer exported and
therefore empty. Fix this by running make twice, once with DUMP=1 and
once without.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-25 23:36:48 +08:00
Tianling Shen
903bd12b2b
scripts/download.pl: add self-hosted mirror
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-23 23:32:35 +08:00
Tianling Shen
1a8c4210ee
Merge Mainline
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-23 18:11:53 +08:00
Tianling Shen
49e3204f02
Revert "treewide: add mirrors for GitHub Codeload"
This reverts commit 9796619efc.

Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-23 16:58:11 +08:00
Tianling Shen
66a13ba4b4
Revert "treewide: add mirrors for GitHub Archives"
This reverts commit 19980fd5b9.

Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-23 16:58:11 +08:00
Tianling Shen
c1f500e1bd
Merge Mainline
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-23 09:29:58 +08:00
Tianling Shen
a53b7d6d62
scripts/download.pl: add download mirrors hosted by SuLingGG
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-22 23:21:13 +08:00
Tianling Shen
46991afe7c
Merge Mainline
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-22 22:06:22 +08:00
Tianling Shen
9f898e3336
download.pl: fix path to mirror 182.140.223.146
Closes: #327

Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-22 17:10:15 +08:00
Paul Spooren
e2ce5345c8 build,json: fix duplicates in default_packages
Calling without the DUMP=1 argument causes the target specific Makefile
to be "included" again which adds the target specific packages twice,
once on the actual run and once included from `include/target.mk`.

This led to duplicate package entries, causing confusion in downstream
projects using the generated JSON files.

While at it, apply `black` style to Python script.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-21 23:36:45 +08:00
Tianling Shen
bca6199163
Merge Mainline
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-19 17:31:52 +08:00
Daniel Golle
d6476932fa include/image*: add support for device-tree overlays
Add new target feature 'dt-overlay' which makes DTC keep the symbol
names in the generated dtb.
Make sure additional DT overlay sources specified by the new device
variable DEVICE_DTS_OVERLAY get compiled together with the main DTS
(currently overlays got to be in the same folder). Let Build/fit pass
the generated DT overlay blobs to mkits.sh.

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
2021-03-18 23:23:45 +08:00
Daniel Golle
82546f9d06 scripts/mkits.sh: add support for adding DT overlay blobs to image
Allow adding multiple device tree overlay blobs to an image and
generate configurations for each of them.
This is useful on boards with modern U-Boot which allow e.g. user-
configurable peripherals ("shields") in that way.
Note that currently, each generated configuration adds exactly one
overlay on top of the base image, ie. adding multiple overlays at the
same time is not yet supported.

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
2021-03-18 23:18:54 +08:00
Paul Spooren
c405955a41 scripts,ipkg-build: use realpath for pkg_dir
This allows manual execution of the ipkg-build script even with
releative path.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-16 11:13:58 +08:00
Paul Spooren
df8b658cb5 scripts,ipkg-build: apply shellcheck
This commit cleans the `ipkg-build` script via changes suggested by
shellcheck. These are mostly word splitting issues.

Remove the definition of GZIP, this adds three "lookups" of the `gzip`
binary but the rest of the build system doesn't seem to use such
improvements neither.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-03-16 11:13:15 +08:00
Tianling Shen
1934e98da6
Merge Mainline
Signed-off-by: Tianling Shen <cnsztl@gmail.com>
2021-03-15 17:51:31 +08:00
Tianling Shen
19980fd5b9
treewide: add mirrors for GitHub Archives
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-15 15:01:44 +08:00
Tianling Shen
9796619efc
treewide: add mirrors for GitHub Codeload
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-15 14:17:10 +08:00
Tianling Shen
93dfa53568
scripts/download.pl: add global mirror for mainland China users
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-15 11:19:22 +08:00
Tianling Shen
fbbdf92ffb
Merge Mainline
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-12 23:28:32 +08:00
Tianling Shen
99a9f78ee3
treewide: adjust tencent mirror
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-11 14:57:01 +08:00
Robert Marko
5e646f442d scripts: mkits.sh: replace @ with - in nodes
U-boot will reject the nodes with @ for the address since
commit:
79af75f777

This in turn will cause the failure to boot with OpenWrt
generated images.

So, to rectify that simply replace @ with -.

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
(cover also newly added rootfs@1 and initrd@1 nodes)
Signed-off-by: Robert Marko <robert.marko@sartura.hr>
2021-03-06 12:19:26 +08:00
Daniel Golle
ab28d07c3b
treewide: fix spelling 'seperate' -> 'separate'
This popular spelling mistake was also introduced by myself lately.
Fix it everywhere.

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-02 22:26:41 +08:00
Daniel Golle
282d5852b6
image: allow building FIT and uImage with ramdisk
Instead of embedding the initrd cpio archive into the kernel, allow
for having an external ramdisk added to the FIT or uImage.
This is useful to overcome kernel size limitations present in many
stock bootloaders, as the ramdisk is then loaded seperately and doesn't
add to the kernel size. Hence we can have larger ramdisks to host ie.
installers with all binaries to flash included (or a web-based
firmware selector).
In terms of performance and total size the differences are neglectible.

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-02 22:24:04 +08:00
Daniel Golle
36c74de2f5
image: add support for building FIT image with filesystem
Allow for single (external-data) FIT image to hold kernel, dtb and
squashfs. In that way, the bootloader verifies the system integrity
including the rootfs, because what's the point of checking that the
hash of the kernel is correct if it won't boot in case of squashfs
being corrupted? Better allow bootloader to check everything needed
to make it at least up to failsafe mode. As a positive side effect
this change also makes the sysupgrade process on nand potentially
much easier as it is now.
In short: mkimage has a parameter '-E' which allows generating FIT
images with 'external' data rather than embedding the data into the
device-tree blob itself. In this way, the FIT structure itself remains
small and can be parsed easily (rather than having to page around
megabytes of image content). This patch makes use of that and adds
support for adding sub-images of type 'filesystem' which are used to
store the squashfs. Now U-Boot can verify the whole OS and the new
partition parsers added in the Linux kernel can detect the filesystem
sub-images, create partitions for them, and select the active rootfs
volume based on the configuration in FIT (passing configuration via
device tree could be implemented easily at a later stage).

This new FIT partition parser works for NOR flash (on top of mtdblock),
NAND flash (on top of ubiblock) as well as classic block devices
(ie. eMMC, SDcard, SATA, NVME, ...).
It could even be used to mount such FIT images via `losetup -P` on a
user PC if this patch gets included in Linux upstream one day ;)

Signed-off-by: John Crispin <john@phrozen.org>
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-02 22:17:37 +08:00
Tianling Shen
9a30cf122a
Merge Mainline
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
2021-03-02 18:19:46 +08:00
David Bauer
faf18cc0c6 download: add mirror alias for Debian
Add an alias for Debian packages and download them from the Debian
mirror redirector.

Signed-off-by: David Bauer <mail@david-bauer.net>
2021-02-27 12:11:33 +08:00
David Bauer
0dd90b05f8 download: use mirror redirector for GNOME downloads
Use the GNOME mirror redirector as the primary download source for GNOME
packages.

Signed-off-by: David Bauer <mail@david-bauer.net>
2021-02-27 12:10:16 +08:00
AmadeusGhost
40ae41824c Merge Mainline 2021-02-25 12:25:05 +08:00
David Bauer
07d3b6e2b8 download: remove broken mirrors
These mirrors don't exist anymore. Remove them.

Signed-off-by: David Bauer <mail@david-bauer.net>
2021-02-22 12:08:16 +08:00
Tianling Shen
6049f2576e
Merge Mainline
Signed-off-by: Tianling Shen <cnsztl@project-openwrt.eu.org>
2021-02-16 14:50:17 +08:00
Felix Fietkau
67004d8112 Mostly revert "build: add support for fixing up library soname"
This reverts commit b12288fa69.
The patchelf approach is too fragile, and the only users of this have been
converted to make patching unnecessary
Leave the abi_version_str variable in place in rules.mk

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2021-02-16 12:13:08 +08:00
Felix Fietkau
8c6c7565ac build: drop ABI version from metadata
Preparation for supporting dynamic ABI versions that depend on the runtime
configuration. Read the suffix from the staging dir pkginfo version files.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2021-02-15 12:17:04 +08:00
Felix Fietkau
e27f09d11f build: add support for fixing up library soname
This makes it possible to declare a package ABI_VERSION independent from the
upstream soname by setting PKG_ABI_VERSION in the package makefile.
The library filename is fixed up for files installed to packages and to the
staging dir. References to the original from executables within the same
package are also fixed up

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2021-02-15 12:16:15 +08:00
AmadeusGhost
5a34d8d677 Merge Mainline 2021-02-10 12:28:03 +08:00
Paul Spooren
0bc6a69776 treewide: unify OpenWrt hosted source via @OPENWRT
Multiple sources are hosted on OpenWrts source server only. The source
URLs to point to the server vary based on different epochs in OpenWrts
history.

Replace all by @OPENWRT which is an "empty" mirror, therefore using the
fallback servers sources.cdn.openwrt.org and sources.openwrt.org.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-02-08 12:11:32 +08:00
CN_SZTL
5a16fd5948
Merge Mainline
Signed-off-by: CN_SZTL <cnsztl@project-openwrt.eu.org>
2021-01-29 00:10:05 +08:00
Paul Spooren
bf200c0467
scripts: sources CDN as fallback in download.pl
In case the default sources for a package fail use the CDN rather than
our own mirror. In case the CDN fails, fallback to our mirror.

Also remove mirror1 which isn't available anymore.

Signed-off-by: Paul Spooren <mail@aparcar.org>
Signed-off-by: CN_SZTL <cnsztl@project-openwrt.eu.org>
2021-01-28 19:40:14 +08:00
CN_SZTL
4ceedc3f17
Merge Mainline
Signed-off-by: CN_SZTL <cnsztl@project-openwrt.eu.org>
2021-01-16 18:13:19 +08:00
Paul Spooren
47c5f0c98f scripts: target-metadata don't add PROFILES twice
Since 4ee3cf2b5a profiles with alternative vendor names may appear
multiple times in `tmp/.targetinfo` or `.targetinfo` (for
ImageBuilders).

The `target-metadata.pl` script adds these profiles then twice to
`PROFILE_NAMES` and the ImageBuilder show the profile twice when running
`make info`.

This patch removes duplicate profile IDs and only adds them once to
`.profiles.mk`.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-01-16 12:05:05 +08:00
CN_SZTL
8dbe7b1459
Merge Mainline 2021-01-07 10:00:33 +08:00
Paul Spooren
027f810151
scripts/feed: no warn on toolchain/linux overwrite
The recent 7f285d "scripts/feeds: warn when skipping core package
override" floods SDK output with warning of overwriting "linux" and
"toolchain" core packages. This should be ignored as these are not
regular packages added via feeds.

While at it slightly improve the warning string.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-01-07 09:57:10 +08:00
Kevin Darbyshire-Bryant
1146051a20 build: drop clang wrapper
clang's gcc emulation does the right thing with -print-file-name now,
drop the wrapper

Signed-off-by: Kevin Darbyshire-Bryant <ldir@darbyshire-bryant.me.uk>
2021-01-06 17:28:08 +08:00
Tianling Shen
fb6cc94a68
Merge Mainline 2021-01-02 12:13:39 +08:00
Paul Spooren
486d2c0968
scripts/qemustart: use squashfs instead of ext4
The qemustart script currently picks the ext4 filesystem rather than
squashfs, while the latter is default for nearly all OpenWrt targets.

Change the default behaviour of qemustart to be in line with the rest.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2021-01-02 12:12:11 +08:00
AmadeusGhost
a47f1a5116 Merge Mainline 2020-12-25 17:25:32 +08:00
Sven Eckelmann
c572423257 ath79: Add support for Plasma Cloud PA300
Device specifications:

* Qualcomm/Atheros QCA9533 v2
* 650/600/217 MHz (CPU/DDR/AHB)
* 64 MB of RAM
* 16 MB of SPI NOR flash (mx25l12805d)
  - 2x 7 MB available; but one of the 7 MB regions is the recovery image
* 2x 10/100 Mbps Ethernet
* 2T2R 2.4 GHz Wi-Fi
* multi-color LED (controlled via red/green/blue GPIOs)
* 1x GPIO-button (reset)
* external h/w watchdog (enabled by default)
* TTL pins are on board (arrow points to VCC, then follows: GND, TX, RX)
* 2x fast ethernet
  - eth0
    + Label: Ethernet 1
    + 24V passive POE (mode B)
    + used as WAN interface
  - eth1
    + Label: Ethernet 2
    + 802.3af POE
    + builtin switch port 2
    + used as LAN interface
* 12-24V 1A DC
* internal antennas

Flashing instructions:

The tool ap51-flash (https://github.com/ap51-flash/ap51-flash) should be
used to transfer the factory image to the u-boot when the device boots up.

Signed-off-by: Sven Eckelmann <sven@narfation.org>
2020-12-23 11:50:07 +08:00
Sven Eckelmann
f109cee0a3 images: Fix sysupgrade.tar for devices with NOR flash
The NOR flash rootfs images stored in a sysupgrade.tar must end with the
JFFS2 marker. Otherwise, devices like OpenMesh A42/A62 are not able to
calculate the md5sum of the fixed squashfs part and store it inside the
u-boot-env.

But the commit ee76bd11bb ("images: fix boot failures on NAND with small
sub pages") adds up to 1020 0x00 bytes after the 0xdead0de EOF marker. The
calculated md5sum will be wrong due do this change and u-boot will fail to
boot the newly flashed device with a message like:

  Validating MD5Sum of 'vmlinux'...
  Passed!
  Validating MD5Sum of 'rootfs'...
  Failed!
      583a1b7b54b8601efa64ade42742459b != 8850ee812dfd7638e94083329d5d2781

  Data validation failed!

and boot the old image again.

Since the original change should not change the behavior of NOR images,
just check for the deadc0de marker at the end of the squashfs-jffs2 image
do avoid the problematic behavior for these images.

Fixes: ee76bd11bb ("images: fix boot failures on NAND with small sub pages")
Signed-off-by: Sven Eckelmann <sven@narfation.org>
2020-12-23 11:48:18 +08:00
Sven Eckelmann
5d58fd6ba3 scripts/om-fwupgradecfg-gen.sh: Generate checksum over whole squashfs
The rootfs is padded to the full block size by padjffs2 and a 4 byte magic
value ("deadc0de") is added to the end. On first boot, the JFFS2 is
replacing the "deadc0de" marker when the rootfs_data is initialized.

The static part of the rootfs is therefore $rootfs_size - 4 and not
$rootfs_size - 262144 - 4.

Signed-off-by: Sven Eckelmann <sven@narfation.org>
2020-12-23 11:38:38 +08:00
Sven Eckelmann
01de8b1389 scripts/om-fwupgradecfg-gen.sh: Drop block alignment code
The padding and block alignment is handled by the image build script and
doesn't need to be duplicated in the fwupgrade.cfg build script.

Signed-off-by: Sven Eckelmann <sven@narfation.org>
2020-12-23 11:38:06 +08:00
INAGAKI Hiroshi
c399be98fd scripts: add -N option to mkhash for printing without newline
Added "-N" option, it allow printing hash(es) without newline.

Signed-off-by: INAGAKI Hiroshi <musashino.open@gmail.com>
2020-12-23 11:34:20 +08:00
CN_SZTL
d9b3d71f34
Merge Mainline 2020-12-19 21:15:16 +08:00
Paul Spooren
0be03639d9
build/json: add filesystem information
Some images are created using different filesystems, most popular
squashfs and ext4. To allow downstream projects to distinguesh between
those, add the `filesystem` information to created json files.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-12-19 18:04:46 +08:00
CN_SZTL
f52579b1b7
Merge Mainline 2020-12-13 10:43:36 +08:00
Karel Kočí
2d44286509 scripts/feeds: fix preference of package install
The previous behavior prefered same feed for dependent packages as
initial package. This caused inconsitency in installation of packages.
The difference was if two feeds provide same package (different version)
there was different result if you executed install for that specific
version compared to install for package depending on it from different
feed.

This ensures that preferred feed is propagated without change and
selected feed is used only really for package it was selected for.

Signed-off-by: Karel Kočí <karel.koci@nic.cz>
2020-12-12 11:35:46 +08:00
CN_SZTL
0ab70179e4
Merge Mainline 2020-12-11 20:13:52 +08:00
Brian Norris
a76ee70a14 scripts/feeds: warn when skipping core package override
Otherwise, a n00b like myself can get quite confused when moving a
package from core to feeds, for example.

(Hint: one *really* needs to clear out the tmp/info/.packageinfo...
entries for the stale package, but '-f' works as well.)

Signed-off-by: Brian Norris <computersforpeace@gmail.com>
2020-12-10 11:17:45 +08:00
CN_SZTL
0f7e04b681
Merge Mainline 2020-11-28 19:17:32 +08:00
Piotr Stefaniak
8fae71281e
build: mkhash on FreeBSD
Apply patch from
https://bugs.openwrt.org/index.php?do=details&task_id=971
in order to make it easier to build OpenWRT on FreeBSD.

Signed-off-by: Piotr Stefaniak <pstef@freebsd.org>
2020-11-28 19:14:14 +08:00
Petr Štetiar
96ab55bf76
download.pl: properly cleanup intermediate .hash file
It seems like after a build the /dl dir seems to now contain a .hash
file for each source file due to inproper cleanup so fix it by removing
those intermediate files before leaving the download action.

Fixes: 4e19cbc553 ("download: handle possibly invalid local tarballs")
Reported-by: Hannu Nyman <hannu.nyman@iki.fi>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2020-11-28 19:14:00 +08:00
Petr Štetiar
603d5f21ab
download: handle possibly invalid local tarballs
Currently it's assumed, that already downloaded tarballs are always
fine, so no checksum checking is performed and the tarball is used even
if it might be corrupted.

From now on, we're going to always check the downloaded tarballs before
considering them valid.

Steps to reproduce:

 1. Remove cached tarball

   rm dl/libubox-2020-08-06-9e52171d.tar.xz

 2. Download valid tarball again

   make package/libubox/download

 3. Invalidate the tarball

   sed -i 's/PKG_MIRROR_HASH:=../PKG_MIRROR_HASH:=ff/' package/libs/libubox/Makefile

 4. Now compile with corrupt tarball source

   make package/libubox/{clean,compile}

Signed-off-by: Petr Štetiar <ynezz@true.cz>
2020-11-28 19:08:14 +08:00
Hannu Nyman
6f8a306caf
scripts/feeds: silence git warning by selecting pull style
Silence the warning in git 2.27 about undefined fast-forward style
in git pull. Define "ff-only" as the style.

Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi>
2020-11-28 19:07:10 +08:00
Paul Spooren
8836fd3d3c
scripts: add size_compare.sh
As package size changes are a continuous topic on the mailing list this
scripts helps developers to compare their local package modifications
against latest upstream.

The script downloads the latest package indexes based on env variables
or the `.config` file. The script compares the actual installed size
(data.tar.gz) or the IPK package size.

An example output is found below:

```
user@dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh
Compare packages of ath79/tiny/mips_24kc:
dropbear busybox iw ubus

Checking configuration difference
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   554  100   554    0     0    336      0  0:00:01  0:00:01 --:--:--   336

 --- start config diff ---
--- /tmp/config.DDjwVh-LOCAL	2020-11-23 09:08:28.913203068 -1000
+++ /tmp/config.DDjwVh-UPSTREAM	2020-11-23 09:08:36.369240887 -1000
@@ -1,5 +1,9 @@
+CONFIG_ALL_KMODS=y
+CONFIG_ALL_NONSHARED=y
 CONFIG_AUTOREBUILD=y
+CONFIG_AUTOREMOVE=y
--- 8< ---
 CONFIG_BINARY_FOLDER=""
+CONFIG_BUILDBOT=y
+CONFIG_TARGET_ALL_PROFILES=y
 CONFIG_TARGET_ROOTFS_DIR=""
 CONFIG_USE_SSTRIP=y
 CONFIG_USE_UCLIBCXX=y
 --- end config diff ---

Checking installed size

Fetching latest package indexes...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 80634  100 80634    0     0  33499      0  0:00:02  0:00:02 --:--:-- 33485
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54082  100 54082    0     0  24252      0  0:00:02  0:00:02 --:--:-- 24252

Comparing package sizes...
Change 	Local	Remote 	Package
+271	51386	51115	base-files
+123	705241	705118	bnx2-firmware
+86	17209	17123	fstools
+22	47989	47967	procd
+21	208311	208290	busybox
+19	67181	67162	netifd

```

I plan to integrate this script into the CI so we have a summary how
sizes change over different architectures.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-28 18:27:07 +08:00
CN_SZTL
e830eb0376
Merge Mainline 2020-11-21 18:58:08 +08:00
David Bauer
506fdee1ed
scripts: download.pl: retry download using filename
With this commit, the download script will try downloading source files
using the filename instead of the url-filename in case the previous
download attempt using the url-filename failed.

This is required, as the OpenWrt sources mirrors serve files using the
filename files might be renamed to after downloading. If the original
mirror for a file where url-filename and filename do not match goes
down, the download failed prior to this patch.

Further improvement can be done by performing this only for the
OpenWrt sources mirrors.

Signed-off-by: David Bauer <mail@david-bauer.net>
2020-11-21 18:02:13 +08:00
Adrian Schmutzler
94a8439922
scripts: add const_structs.checkpatch for checkpatch.pl
Kernel has separated the structs that are reported to be const in
checkpatch.pl into a file of its own, const_structs.checkpatch.

This file has been missing after the recent update of checkpatch.pl,
leading to the following message:

  No structs that should be const will be found - file
  '/data/openwrt/scripts/const_structs.checkpatch': No such file
  or directory

This commit adds the relevant file from v5.10-rc4.

Fixes: 086ee09bbc ("scripts: Update checkpatch.pl to 2020-06-11")

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2020-11-21 17:58:30 +08:00
Adrian Schmutzler
e7234618b8
scripts: checkpatch.pl: suppress warnings about MAINTAINERS
The kernel expects changes to MAINTAINERS for all removed or added
files, printing warnings like:

  WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
  #828:
  deleted file mode 100644

Since this does not apply to "our" files in OpenWrt repo, this
warning should be disabled.

This can be achieved easiest by setting $reported_maintainer_file
to 1. While this is a hack that tricks the script into believing
the proper MAINTAINERS changes have been made, it's the easiest
solution as it does not require to touch any other code.

Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
2020-11-21 17:58:25 +08:00
CN_SZTL
d2489e6aeb
Merge Mainline 2020-11-15 00:49:56 +08:00
Paul Spooren
b7db313bc8
build: store SOURCE_DATE_EPOCH in JSON info files
The source date epoch is the only reproducible date close to the actual
build date. It can be used for tooling like the firmware wizard to show
the image age.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-14 23:34:48 +08:00
Paul Spooren
a7b5fd56ef
build,json: fix compatibility with Python 3.5
The f-string feature was introduced in Python 3.6. As Buildbots may run
on Debian 9, which comes per default with Python 3.5, this would cause
an issue. Instead of f-strings use the *legacy* `.format()` function.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-14 23:34:21 +08:00
Huangbin Zhan
ef690c4d36
build: fix compatibility with python 3.6
On a system python3 is linked to python3.6, fail to perform json_overview_image_info
 and got `TypeError: __init__() got an unexpected keyword argument 'capture_output'`.
This patch emulate the behaviour on python 3.7+.

Signed-off-by: Huangbin Zhan <zhanhb88@gmail.com>
2020-11-14 23:33:47 +08:00
Daniel Golle
15c4e9d88c
build,json: fix build failure in case no data is found
Only collect arch_packages if actually generating any output.

Fixes: commit f09b9319 ("build,json: store arch_packages in profiles.json"(
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-14 23:33:30 +08:00
Paul Spooren
6e6e62ac5b
build,json: store arch_packages in profiles.json
The `arch_packages` contains the supported package architecture.

Previously it was necessary to parse the `Packages` index for the line
`Architecture:`, requiring both an additional parser and file download.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-14 23:33:13 +08:00
Paul Spooren
4829b65c29
scripts: JSON merge don't crash if no JSON found
The JSON `WORK_DIR` ($(KDIR)/json_info_files) is only created if the new
image generation methods from `image.mk` are used. However some targets
like `armvirt` do not use it yet, so the folder is never created.

The `json_overview_image_info.py` script used to raise an error if the
given `WORK_DIR` isn't a folder, however it should just notify about
missing JSON files.

This patch removes the Python assert and exists with code 0 even if no
JSON files were found, as this is not necessarily an error but simply
not yet implemented. Using `glob` on an not existing `Path` results in
an empty list, therefore the for loop won't run.

Signed-off-by: Paul Spooren <mail@aparcar.org>
CC: Petr Štetiar <ynezz@true.cz>
2020-11-14 23:32:14 +08:00
Paul Spooren
d871bcd6d5
build: store default/device packages in JSON
With this commit the `profiles.json` contain both the target specific
`default_packages` as well as the device specific `device_packages` as a
array of strings.

This information is required for downstream projects like the various
web-based interactive firmware generators.

Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-14 23:30:40 +08:00
Paul Spooren
f20e414b79
build: refactor JSON info files to profiles.json
JSON info files contain machine readable information of built profiles
and resulting images. These files were added in commit 881ed09ee6
("build: create JSON files containing image info").

They are useful for firmware wizards and script checking for
reproducibility.

Currently all JSON files are stored next to the built images, resulting
in up to 168 individual files for the ath79/generic target.

This patch refactors the JSON creation to store individual per image
(not per profile) files in $(BUILD_DIR)/json_info_files and create an
single overview file called `profiles.json` in the target directory.

Storing per image files and not per profile solves the problem of
parallel file writes. If a profiles sysupgrade and factory image are
finished at the same time both processes would write to the same JSON
file, resulting in randomly broken outputs.

Some target like x86/64 do not use the image code yet, resulting in
missing JSON files. If no JSON info files were created, no
`profiles.json` files is created as it would be empty anyway.

As before, this creation is enabled by default only if `BUILDBOT` is set.

Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64.

Signed-off-by: Paul Spooren <mail@aparcar.org>
[json_info_files dir handling in Make, if case refactoring]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2020-11-14 23:29:53 +08:00
CN_SZTL
604ea1e87f
Merge Mainline 2020-11-14 13:55:18 +08:00
Felix Fietkau
39a4504834
scripts/kconfig.pl: allow regex syntax in filtering out config entries
This will be used to filter out some autogenerated config values from
the kernel config files

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2020-11-14 13:51:51 +08:00
Paul Spooren
261799f6d6
scripts: add spelling.txt for checkpatch.pl
Running the updated checkpatch version with spelling.txt show that
spelling mistakes happen to everyone.

> /target/linux$ fd .*.patch | xargs ../../scripts/checkpatch.pl | rg spell
WARNING: 'usefull' may be misspelled - perhaps 'useful'?
WARNING: 'afecting' may be misspelled - perhaps 'affecting'?
WARNING: 'usefull' may be misspelled - perhaps 'useful'?
WARNING: 'afecting' may be misspelled - perhaps 'affecting'?
WARNING: 'begining' may be misspelled - perhaps 'beginning'?
WARNING: 'superflous' may be misspelled - perhaps 'superfluous'?
WARNING: 'multipe' may be misspelled - perhaps 'multiple'?
WARNING: 'recieves' may be misspelled - perhaps 'receives'?
WARNING: 'retreive' may be misspelled - perhaps 'retrieve'?
WARNING: 'tranfer' may be misspelled - perhaps 'transfer'?
WARNING: 'additonal' may be misspelled - perhaps 'additional'?
WARNING: 'accomodate' may be misspelled - perhaps 'accommodate'?
[...]

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-14 13:45:59 +08:00
Paul Spooren
293e8623bb
scripts: checkpatch remove device-tree doc check
Remove checks for device tree documentation as the OpenWrt tree comes
withouth the ./Documentation folder.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-14 13:45:51 +08:00
Paul Spooren
fe0b1bfe0c
scripts: Update checkpatch.pl to 2020-06-11
Checkpatch grown in functionallity and we should make use of that. If
OpenWrt patches should be upstream material they should also be checked
based on upstream checkpatch.pl instead of 2013.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-14 13:45:00 +08:00
CN_SZTL
38f7610841
Merge Mainline 2020-11-12 15:44:23 +08:00
Paul Spooren
d5c27b31ff
scripts: mkhash fail on hashing a folder
mkhash currently returns the hash of an empty input when trying to hash
a folder. This can be missleading in caseswhere e.g. an env variable is
undefined which should contain a filename. `mkhash ./path/to/$FILE`
would exit with code 0 and return a legit looking checksum.

A better behaviour would be to fail with exit code 1, which imitates the
behaviour of `md5sum` and `sha256sum`.

To avoid hashing of folders the `stat()` is checked.

Hashing empty inputs result in the following checksums:
md5: d41d8cd98f00b204e9800998ecf8427e
sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-12 11:10:10 +08:00
Paul Spooren
31329b5a30
scripts: mkhash fix return code handling
If hashing a file fails mkhash shouldn't just silently fail. Now check
after each call of `hash_file()` the return and exit early in case of
errors. The return value which was previously ignored and would always
return 0.

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-12 11:10:01 +08:00
Paul Spooren
d919b06594
scripts: mkhash show -n option in usage
The -n option prints the filename of hashed files next to the calculated
checksum. Reflect that in the usage message.

user@dawn:~/src/openwrt/openwrt$ ./a.out md5 -n .config
eb06db36e7b6751cb18801945e46bf5d .config

Signed-off-by: Paul Spooren <mail@aparcar.org>
2020-11-12 11:09:53 +08:00
CN_SZTL
7e654c67ed
Merge Mainline 2020-11-08 00:19:27 +08:00
Petr Štetiar
9a187d8f24 scripts: bundle-libraries.sh: fix broken SDK compiler
Recent versions (> 5.33) of `file` report liblto_plugin.so as
executable:

 $ file liblto_plugin.so
 liblto_plugin.so.0.0.0: ELF 64-bit LSB pie executable ...

Which then leads to improper packaging of the plugin, resulting in the
broken compiler:

 configure: checking whether the C compiler works
 mips-openwrt-linux-musl/bin/ld: liblto_plugin.so: error loading plugin: liblto_plugin.so: invalid ELF header

As the LTO compiler plugin library is incorrectly packaged as SDK
executable:

 $ head -1 ~/staging_dir/toolchain...libexec/gcc/.../liblto_plugin.so
 #!/usr/bin/env bash

Fix this by filtering out shared libraries from the patching.

Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1296868
Acked-by: Jo-Philipp Wich <jo@mein.io>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2020-11-03 11:54:47 +08:00
Petr Štetiar
df8e43981c scripts/getver.sh: silence rev-list errors
Shallow copies are quite common on CI platforms nowadays, making REBOOT
tag unavailable, thus producing following confusing errors in the build
logs:

 fatal: Invalid revision range ee53a240ac902dc83209008a2671e7fdcf55957a..HEAD
 fatal: Invalid revision range ee53a240ac902dc83209008a2671e7fdcf55957a..0493d57e04774d47921a7d2014b567455d5dc16b

Signed-off-by: Petr Štetiar <ynezz@true.cz>
2020-11-03 11:53:49 +08:00