Skip to main content ↑ ↓
Topic: SOLVED: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64 (Read 2090 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

SOLVED: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

I have a running armtix system using the 'official' rpi boot process; my init is runit.

I upgraded from linux-rpi-5.15.33-2-aarch64 to linux-rpi-5.15.34-2-aarch64 and the system refuses to proceed after most of the boot is completed it clears the terminal and then hangs. For some reason it seems as though the system is reading or writing the sd card as the green light is permanently on. I tried ctl+alt+f2 to obtain a terminal, but no luck,

To try and debug I used another PC (fsck showed no errors on the card). I changed the current services link to single which contains just sulogin.  Again the boot failed.

Back on the main PC I reinstated /boot and /usr/lib/modules/5.15.33-2-rpi-ARTIX from the previous pkg file.

On reboot the system works fine so I think the problem must lie in the latest kernel somewhere.

I looked at the armtix cmdline.txt & config.txt, but they had not changed so I don't know if I need to change my versions of those files.

Any ideas how to try and debug this issue?
Code: [Select]
$ diff -r /tmp/boot/ /boot/
Binary files /tmp/boot/initramfs-linux-rpi4.img and /boot/initramfs-linux-rpi4.img differ
Binary files /tmp/boot/kernel8.img and /boot/kernel8.img differ
Binary files /tmp/boot/overlays/mipi-dbi-spi.dtbo and /boot/overlays/mipi-dbi-spi.dtbo differ
Binary files /tmp/boot/rpi4Image and /boot/rpi4Image differ

Re: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #1 –
Did you try to catch logs from UART?
ARMtix

Re: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #2 –
I can certainly try. I have a uart connection on another rpi and a spare serail usb cord somewhere. I will have a go; I looked at dmesg.log and it seems to be overwritten on each boot.

Re: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #3 –
Quote from: phoenix_king_rus – on
Did you try to catch logs from UART?
hi phoenix_king_rus I have two large logs from serial  boots of both systems; the 5.15.34-2 boot ends in a kdb section which I don't know how to proceed with.

Most of the log is the same here is the kdb output

Code: [Select]
[   12.416712] bcm2835-isp bcm2835-isp: Loaded V4L2 bcm2835-isp
[   12.428669] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[   12.511171] vc4-drm gpu: bound fe600000.firmwarekms (ops vc4_fkms_ops [vc4])
[   12.518888] usbcore: registered new interface driver brcmfmac
[   12.518996] [drm] Initialized vc4 0.0.0 20140616 for gpu on minor 1
[   12.544115] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.bin failed with error -2

Entering kdb (current=0xffffff8100370000, pid 451) on processor 3 Oops: (null)
due to oops @ 0xffffffdfde26ce08
CPU: 3 PID: 451 Comm: udevd Tainted: G         C        5.15.34-2-rpi-ARTIX #1
Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : vc4_atomic_commit_tail+0x38/0x780 [vc4]
lr : commit_tail+0xa0/0x180 [drm_kms_helper]
sp : ffffffc009d6b210
x29: ffffffc009d6b210 x28: ffffff810a8a2ab0 x27: 0000000000000000
x26: ffffff81025f2800 x25: 0000000000000038 x24: ffffff810a886000
x23: 0000000000000000 x22: ffffffdfde29b4c8 x21: ffffff810a886000
x20: ffffff810a948480 x19: 0000000000000000 x18: 0000000000000000
x17: 0000000000000000 x16: ffffffe03f71e8a0 x15: 0000000000000000
x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
x8 : ffffff8107c7e300 x7 : 0000000000000000 x6 : 000000002b1a511d
x5 : 0000000000000001 x4 : 0000000000000001 x3 : ffffff810a948538
x2 : 0000000000000000 x1 : ffffff8100370000 x0 : 0000000000000000
Call trace:
 vc4_atomic_commit_tail+0x38/0x780 [vc4]
 commit_tail+0xa0/0x180 [drm_kms_helper]
 drm_atomic_helper_commit+0x15c/0x370 [drm_kms_helper]
 drm_atomic_commit+0x4c/0x5c [drm]
 drm_client_modeset_commit_atomic+0x1c8/0x260 [drm]
 drm_client_modeset_commit_locked+0x5c/0x1a0 [drm]
 drm_client_modeset_commit+0x30/0x60 [drm]
 drm_fb_helper_set_par+0xc8/0x120 [drm_kms_helper]
 fbcon_init+0x39c/0x4cc
 visual_init+0xb4/0x104
 do_bind_con_driver.isra.0+0x1c8/0x3a0
 do_take_over_console+0x140/0x200
 do_fbcon_takeover+0x6c/0xe0
 fbcon_fb_registered+0x100/0x11c
 register_framebuffer+0x1f4/0x320
 __drm_fb_helper_initial_config_and_unlock+0x338/0x534 [drm_kms_helper]
 drm_fbdev_client_hotplug+0x144/0x224 [drm_kms_helper]
 drm_fbdev_generic_setup+0xb4/0x190 [drm_kms_helper]
 vc4_drm_bind+0x248/0x300 [vc4]
 try_to_bring_up_master+0x164/0x1d0
 component_master_add_with_match+0xb0/0x100
 vc4_platform_drm_probe+0xc0/0x100 [vc4]
 platform_probe+0x68/0xe0
 really_probe.part.0+0x9c/0x30c
 __driver_probe_device+0x98/0x144
 driver_probe_device+0xc8/0x15c
 __driver_attach+0xf8/0x190
 bus_for_each_dev+0x70/0xd0
 driver_attach+0x24/0x30
 bus_add_driver+0x108/0x1f0
 driver_register+0x78/0x130
 __platform_driver_register+0x28/0x34
 vc4_drm_register+0x48/0x1000 [vc4]
 do_one_initcall+0x44/0x290
 do_init_module+0x48/0x240
 load_module+0x1dd0/0x222c
 __do_sys_init_module+0x12c/0x1c0
 __arm64_sys_init_module+0x1c/0x30
 invoke_syscall+0x48/0x114
 el0_svc_common.constprop.0+0x44/0xfc
 do_el0_svc+0x28/0x90
 el0_svc+0x28/0x80
 el0t_64_sync_handler+0xa4/0x130
 el0t_64_sync+0x1a0/0x1a4

[3]kdb>

the corresponding output in 5.15.33-2 looks like

Code: [Select]
[   16.219108] 8021q: 802.1Q VLAN Support v1.8
[   16.493399] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[   16.503262] bcmgenet fd580000.ethernet eth0: Link is Down
[   16.520790] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled

                   '                      root@pikat
                  'o'                     ----------
                 'ooo'                    OS: Artix Linux aarch64
                'ooxoo'                   Host: Raspberry Pi 4 Model B Rev 1.4
               'ooxxxoo'                  Kernel: 5.15.33-2-rpi-ARTIX
              'oookkxxoo'                 Uptime: 14 secs
             'oiioxkkxxoo'                Packages: 697 (pacman)
            ':;:iiiioxxxoo'               Shell: bash 5.1.16
               `'.;::ioxxoo'              Resolution: 1920x1080
          '-.      `':;jiooo'             Terminal: 2
         'oooio-..     `'i:io'            CPU: BCM2835 (4) @ 1.500GHz
        'ooooxxxxoio:,.   `'-;'           Memory: 89MiB / 7822MiB
       'ooooxxxxxkkxoooIi:-.  `'
      'ooooxxxxxkkkkxoiiiiiji'
     'ooooxxxxxkxxoiiii:'`     .i'
    'ooooxxxxxoi:::'`       .;ioxo'
   'ooooxooi::'`         .:iiixkxxo'
  'ooooi:'`                `'';ioxxo'
 'i:'`                          '':io'
'`                                   `'

pikat login: [   17.927803] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
[   19.816448] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
[   20.608494] bcmgenet fd580000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   20.616768] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   21.007784] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   33.760345] cam-dummy-reg: disabling

I can supply the full files if they are needed; not sure where I should put them.

Re: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #4 –
I tried updating the rpi4 firmware and then the latest kernel 5.15.34-4, but still did not manage to boot. As before reverting to 5.15.33-2 restored the boot process.

My rpi4 firmware
Code: [Select]
$ sudo /usr/bin/rpi-eeprom-update 
BOOTLOADER: up to date
   CURRENT: Tue 25 Jan 14:30:41 UTC 2022 (1643121041)
    LATEST: Tue 25 Jan 14:30:41 UTC 2022 (1643121041)
   RELEASE: critical (/lib/firmware/raspberrypi/bootloader/critical)

  VL805_FW: Using bootloader EEPROM
     VL805: up to date
   CURRENT: 000138a1
    LATEST: 000138a1

Re: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #5 –
That's strange. Have you heard about problems like this on Arch? The kernel configs are the same
ARMtix

Re: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #6 –
Only thing I see that might be related is this

https://github.com/raspberrypi/linux/issues/4145

where they mention the brcmfmac

I noticed that the 5.15.34-4 boot changed somewhat. Although it hung I did not see the disk activity light permanently on. I will try and capture the  serial console to see if the same error is present.

Edit: I got a console dump for  5.15.34-4 and I see the same kind of message regarding firmware, but in changed order.

Re: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #7 –
The bt in kdb shows udevd at the top eg 
Code: [Select]
[2]kdb> bt
Stack traceback for pid 446
0xffffff8103193b80      446      424  1    2   R  0xffffff81031954d0 *udevd
CPU: 2 PID: 446 Comm: udevd Tainted: G         C        5.15.34-4-rpi-ARTIX #1
Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
Call trace:
 dump_backtrace+0x0/0x1a0
and the kdb lsmod shows it to be in the process of loading vc4 and bluetooth

Edit: just in case I did try adding the line dtparam=krnbt=on to confg.txt as it has recently appeared in config.txt.rpi4

Re: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #8 –
What if you try to blacklist vc4/wifi/bluetooth modules?
ARMtix

Re: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #9 –
I have this
Code: [Select]
#uart on pins 14 15
dtoverlay=disable-bt
in config.txt. I was wondering if this was the problem so tried commenting, I don't think that made a difference, but my serial console won't work with that commented. I tried various things to try and get the mini-uart to work, but the various web resources are very confusing and I didn't manage to get it to work and the 5.15.34 certainly didn't boot.
 
Is there a kernel module that can be blacklisted in /etc/modprobe.d/?

Re: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #11 –
Quote from: phoenix_king_rus – on
Try bluetooth, brcmfmac and vc4
Well the crash is prevented if I just blacklist vc4, but I cannot use X11 after that.

My config has this related

Code: [Select]
#enable vc4
dtoverlay=vc4-fkms-v3d
max_framebuffers=2
I will try using  using vc4-fkms-v3d-pi4?
Edit: I tried vc4-fkms-v3d-pi4 and that made no difference, if I don't use either the system boot, but cannot start X11. If I use
Code: [Select]
dtoverlay=vc4-kms-v3d,cma-64
I get a bootable system and X11 will start; it messes up the sound device order though compared with  linux-rpi-5.15.33-2-aarch64 . FWIW
Code: [Select]
$ uname -r
5.15.34-4-rpi-ARTIX

Edit: by messed up  I mean the following
Code: [Select]
$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: Headphones [bcm2835 Headphones], device 0: bcm2835 Headphones [bcm2835 Headphones]
  Subdevices: 8/8
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
  Subdevice #7: subdevice #7
card 1: vc4hdmi0 [vc4-hdmi-0], device 0: MAI PCM i2s-hifi-0 [MAI PCM i2s-hifi-0]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 2: vc4hdmi1 [vc4-hdmi-1], device 0: MAI PCM i2s-hifi-0 [MAI PCM i2s-hifi-0]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
clearly the headphones only have one output device; so the hdmi device(s)  get mixed up with Headphones

SOLVED: Re: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #12 –
FWIW I find that the problem has gone away in 5.15.36-2-rpi-ARTIX-RPI4 so I consider the problem solved. Does the change of kernel name (adding -RPI4) mean something?

Re: SOLVED: rpi4 boot failure upgrading to linux-rpi-5.15.34-2-aarch64

Reply #13 –
Quote from: replabrobin – on
Does the change of kernel name (adding -RPI4) mean something?
It's more about compatibility. At some point i reverted from -ARTIX-RPI4 to -ARTIX but now there is linux-aarch64-lts which may have same version
ARMtix