File System / SSD Failures on Lenovo X250 (not kernel, not hardware)

I've been experiencing very frequent file system / ssd failures on my Lenovo X250. I've experienced this problem:

on two separate, brand new Lenovo x250s
had a Lenovo x250 working with Debian (Jesse) with no issues for months (same ssd, same firmware)
across several kernels: 3.18, 4.0, 4.1, 4.2, and 4.3
with or without TRIM enabled (and all the fstab incanations: noatime,nodirtime,discards, etc.

I am fairly certain this is a configuration issue. Specifically, NixOS likely isn't installing the right modules or setting the right configuration or having an option to grab the right firmware?
I'm really desperate to fix this problem so I can use NixOS at work.
The Issue
It is hard to re-create and hard to debug. There is a kernel failure, the file system will hang, and get remounted in read-only mode (so I can't grab logs). The error happens intermittently. The longest period I've gone without experiencing the issue is a day and a half, but I've experienced it immediately on reboot a few times. It seems to happen more frequently after suspend, but not exclusively (I hit it one right after a reboot - I also hit it once during systemctl rescue.
I have taken photos of the failure on my cell phone (sometimes I can catch logs while tailing dmesg when the error happens).
from dmesg (I'm writing out what I see on my photo):
ata1.00: exception Emask 0x0 SAct 0x1e0000 SErr 0x50000 action 0x6 frozen
ata1: SError: { PHYRdyChg CommWake}
ata1.00: failed command: WRITE FPDMA QUEUED
ata1.00: cmd 61/08:00:18:ee:7a/00:00:02:00:00/40 tag 0 ncq .... (and on)
ata1.00: status: { DRDY }
...
(and repeat)
...
ata1: hard resetting link
ata1: SATA link down (SStatus 0 SControl 300)
...
(repeat)
...
ata1.00: disabled
ata1.00: device reported invalid CHS sector 0
sd 0:0:0:0: [sda] tag#21 FAILED Result: hostbye=DID_OK drivebyte=DRIVER_SENSE
sd 0:0:0:0: [sda] tag#21 Sense Key : Illegal Request [current] [descriptor]
sd 0:0:0:0: [sda] tag#21 AddI Sense: Unaligned write command
sd 0:0:0:0: [sda] tag#21 CDB: Write(10) 2a 00 0f 64 4c 78 00 00 10 00
blk_update_request: I/O error, dev sda, sector 258231416
...
(and repeat with increasing tag # and new blk_update sector errors)
...
EXT4-fs warning (device dm-2): ext4_end_bio:332: I/O error -5 writing to inode 13389888 (offset 0 size 28672)
Buffer I/O Error on device dm-2, logical block 58770479
...
(repeat buffer i/o errors)
...
blk_update_request (as above)
sd 0:0:0:0: rejecting I/O to offline device
...
ata1: BH  complete
...
Abortin journal on device dm-2-8
EXT4-fs (dm-2): Delayed block allocation failed for inode 14600222 at inode ...
EXT4-fs (dm-2): This should not happen!! Data will be lost
...
(and on)

My Setup

Lenovo X250
luks encrypted drive. (setup following these instructions)

☭ uname -a
Linux nixos 4.3.0 #1-NixOS SMP Thu Jan 1 00:00:01 UTC 1970 x86_64 GNU/Linux

☭ nixos-version
15.09.706.45128de (Dingo)

☭ lspci
00:00.0 Host bridge: Intel Corporation Broadwell-U Host Bridge -OPI (rev 09)
00:02.0 VGA compatible controller: Intel Corporation Broadwell-U Integrated Graphics (rev 09)
00:03.0 Audio device: Intel Corporation Broadwell-U Audio Controller (rev 09)
00:14.0 USB controller: Intel Corporation Wildcat Point-LP USB xHCI Controller (rev 03)
00:16.0 Communication controller: Intel Corporation Wildcat Point-LP MEI Controller #1 (rev 03)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (3) I218-LM (rev 03)
00:1b.0 Audio device: Intel Corporation Wildcat Point-LP High Definition Audio Controller (rev 03)
00:1c.0 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #6 (rev e3)
00:1c.1 PCI bridge: Intel Corporation Wildcat Point-LP PCI Express Root Port #3 (rev e3)
00:1d.0 USB controller: Intel Corporation Wildcat Point-LP USB EHCI Controller (rev 03)
00:1f.0 ISA bridge: Intel Corporation Wildcat Point-LP LPC Controller (rev 03)
00:1f.2 SATA controller: Intel Corporation Wildcat Point-LP SATA Controller [AHCI Mode] (rev 03)
00:1f.3 SMBus: Intel Corporation Wildcat Point-LP SMBus Controller (rev 03)
00:1f.6 Signal processing controller: Intel Corporation Wildcat Point-LP Thermal Management Controller (rev 03)
02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5227 PCI Express Card Reader (rev 01)
03:00.0 Network controller: Intel Corporation Wireless 7265 (rev 59)

☭ sudo smartctl -a /dev/sda
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-4.3.0] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     SAMSUNG MZ7LN256HCHP-000L7
Serial Number:    S20HNXBG772065
LU WWN Device Id: 5 002538 d00000000
Firmware Version: EMT03L6Q
User Capacity:    256,060,514,304 bytes [256 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Nov 26 11:02:36 2015 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection: 		(    0) seconds.
Offline data collection
capabilities: 			 (0x53) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 133) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       76
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       151
170 Unknown_Attribute       0x0032   100   100   010    Old_age   Always       -       0
171 Unknown_Attribute       0x0032   100   100   010    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   010    Old_age   Always       -       0
173 Unknown_Attribute       0x0033   099   099   005    Pre-fail  Always       -       1
174 Unknown_Attribute       0x0032   099   099   000    Old_age   Always       -       38
178 Used_Rsvd_Blk_Cnt_Chip  0x0013   100   100   010    Pre-fail  Always       -       0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0013   100   100   010    Pre-fail  Always       -       773
184 End-to-End_Error        0x0033   100   100   097    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0032   077   056   000    Old_age   Always       -       23
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
233 Media_Wearout_Indicator 0x0013   099   099   000    Pre-fail  Always       -       16768826
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       194
242 Total_LBAs_Read         0x0032   099   099   000    Old_age   Always       -       823
249 Unknown_Attribute       0x0032   099   099   000    Old_age   Always       -       256

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

Warning! SMART Selective Self-Test Log Structure error: invalid SMART checksum.
SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
  255        0    65535  Read_scanning was never started
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

You use ext4? I had probably the same issue in April until I've moved to F2FS.
I had the same issue on 2 ssd in raid 0.

@jagajaga yes, ext4. how stable have you found F2FS?

@aaronlevin pretty stable, 0 issues from April. Running on my laptop (intel ssd) and desktop with 2 ssd (ocz) in raid 1 nowadays.

I'm typing this from an X250 with LUKS, LVM and ext4 on SSD. NixOS since day 1, current uptime 29 days. Do you have allowDiscards = true in boot.initrd.luks.devices?

@mbakke I have experienced the issue with allowDiscards = true;. It's currently turned off, so I'll turn it back on again and we'll see how quickly the issue surfaces.
@mbakke what version of the firmware on the ssd are you using? Also, anything special in the kernel modules you're loading or any extra modprobe config?
Thanks!

Hm, I have a different SSD altogether (Toshiba). Nothing else hardware-related in nixos configuration.
Device Model:     TOSHIBA THNSNJ512GCSU
Firmware Version: JULA0101
User Capacity:    512,110,190,592 bytes [512 GB]

Samsung has a pretty poor track record when it comes to Linux SSD firmware.. Can you check if the same happens if you boot with libata.force=noncq as per this bug report?

@mbakke I had the libata.force=noncq in my extraModprobeConfig and still had the issue.

@mbakke interesting that you have a different ssd. However, we have debian (Jesse) installed on a similar model as mine (same ssd, same firmware) and no issues.

@aaronlevin you need to set that in boot.kernelParams, not extraModProbeConfig. I suspect Debian may have blacklisted NCQ on that device.

@mbakke does that go in my configuration.nix or my hardware-configuration.nix?

Either :)

@mbakke ok, perhaps that was my issue: having libata.force=noncq in my extraModProbe and not kernelParams. I've added it and I'll see how stable my ssd is now.
Is there a policy around closing and re-opening? Because I'm happy to close this now and then re-open it if does not fix the issue.
Additionally, should we possibly consider generating that kernel param on detection of this ssd similar to debian? Or is that out of scope for NixOS to determine such a setting?

If that indeed solves the issue, we can probably add the NCQ TRIM blacklist patch from above, at least if other distros are doing the same. Although it arguably should be added upstream...
(disclaimer: I don't actually know what NixOS' stance on adding kernel patches is, nor am I an official dev)

@mbakke hmm, good question. We may not even need to apply the patch. For example, it might even be easier just to generate:
{ config, pkgs, ... }:

{
    # We have detected an SSD with NCQ TRIM blacklisted.
    boot.kernelParams = [ "libata.force=noncq" ];
}

Is there a policy around closing and re-opening? Because I'm happy to close this now and then re-open it if does not fix the issue.

I don't think there's such a policy. I would do the same as you.

@vcunat thanks.

After 2 days of stability (longest so far), I just hit the issue again. Ugh. I was really hoping that would resolve my issue.
To update: I tried putting libata.force=noncq in my kernelParams. This brought about some stability but I hit the issue after 48 hours.

Can anyone think of any other settings that a distro like debian might set for these SSDs that NixOS is not setting?

I don't suppose you were able to take a screendump this time? The errors should be somewhat different. Could perhaps try mounting /var/log/journal on a USB stick, or send it over network. You can also disable NCQ runtime with echo 1 > /sys/block/sda/device/queue_depth.
Remove discard from fstab too to make sure we don't hit multiple bugs. I'm on BIOS 1.15 FWIW, although we should both upgrade to 1.17. Debian follows kernel development closely and may apply all kinds of workarounds that are fixed in newer firmware.

I'm also seeing this issue on a SuperMicro SuperServer 5017R-WRF with Samsung 840 EVO SSDs, both a 250GB and a 500GB model. I'm running NixOS 15.09 and ext4. The same hardware runs Ubuntu 14.04 with no issues.
In my case, the controller resets the drive enough times that it ends up in UDMA/133 mode and appears to be stable, but I've only just managed to get nixos-install to finish, so I haven't taxed it much yet.
I added libata.force=noncq on the NixOS installer grub command line and got fewer errors than without that boot parameter, but still got them.

@mbakke I just want to preface my answers by thanking you for all your help!!
NCQ looks properly disabled and I don't have discards in my fstab (listed below). Is there a simple way to upgrade the firmware?
☭ cat /sys/block/sda/device/queue_depth
1

here is my hardware-configuration.nix:
{ config, lib, pkgs, ... }:

{
  imports =
    [ <nixpkgs/nixos/modules/installer/scan/not-detected.nix>
    ];

  boot.initrd.availableKernelModules = [ "xhci_pci" "ehci_pci" "ahci" "usbhid" "usb_storage" ];
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/bb6a6acb-055e-4d1f-9812-13c9d183bb6c";
      fsType = "ext4";
      options = "rw,relatime,nobarrier,data=ordered";
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/9cbf3855-bb29-4123-abd1-e08de2e39a36";
      fsType = "ext2";
    };

  swapDevices =
    [ { device = "/dev/disk/by-uuid/73bfe4e8-b4a6-433b-b152-73fd5702fcd8"; }
    ];

  nix.maxJobs = 4;
}

@dhess can you run smartctl -a on your drive? Curious what firmware you have running.

@dhess: Your device is explicitly blacklisted in kernel 4.1 or newer. Try setting boot.kernelPackages = pkgs.linuxPackages_4_1 and see if the problem persists. It may be easier to switch to the unstable channel if you need to compile stuff: nix-channel --add https://nixos.org/channels/nixos-unstable nixos.
@aaronlevin if you still have an X250 with Debian around, check if the backports kernel has the same issue (apt-get -t jessie-backports install linux-image-amd64). Jessie and 14.04 are both kernel 3.16, while 15.09 is 3.18. Could you also try booting 4.2 or newer with libata.force=noncqtrim instead of libata.force=noncq.
If you are on stock BIOS still, I recommend upgrading. These are first generation Broadwells after all. Download the (Windows) ISO-based installer (n10ur08w.exe IIRC) from Lenovo, extract the .iso with innoextract, convert to image with geteltorito.pl and dd onto USB stick. Instructions here.

Yes, that appears to be exactly what's happening with the newest Samsung firmware.
Based on Samsung's responses to the bug, I think that, rather than upgrading my kernel, I'll just take my business elsewhere!
Thanks for the help.

@mbakke the Debian system in question was on kernel 4.1, but I will try booting with libata.force=noncqtrim instead and see if the problem persists. I hit the issue this morning, so it's happening with some regular frequency.
I'll also try upgrading the firmware but that might not happen until later this evening.
Thanks again!

PS - I am on the 4.3 kernel.

@mbakke do I need to have "libata" in my boot.initrd.kernelModules? It is not there currently.

@aaronlevin the libata module is loaded automatically when needed. I assumed it was compiled-in, but since it's built as a module the libata.force options should work in extraModProbeConfig too.

@mbakke just to make sure I have everything correct:
in configuration.nix I have:  boot.kernelParams = [ "libata.force=noncqtrim" ];
in hardware-configuration.nix I have:
  boot.initrd.availableKernelModules = [ "xhci_pci" "ehci_pci" "ahci" "usbhid" "usb_storage" ];
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];

So, I'm not explicitly specifying the libata module needs to be loaded, though it appears in my kernelParams. I would assume I should have to specify it in boot.initrd.kernelModules?

You can put libata.force in either kernel command line or as a module parameter, it should be picked up either way. The libata module is loaded automatically (but it's arguably more nix-y to specify it).

Hmm, I'm now getting this same error using an Intel 535 120GB SSD, which is not on the blacklist.

I just hit the issue right now again. I've added "libata" to my boot.initrd.kernelModules and rebooted. we'll see.

I'm back on the Samsung 840 EVO 250GB SSD again. I managed to get the 4.1 kernel installed by putting the SATA controller in IDE mode in the system BIOS. I then rebooted and set the controller back to AHCI mode. Unfortunately, even in 4.1, I'm getting NCQ errors:
Dec 02 10:09:52 nix01 kernel: ata2.00: exception Emask 0x10 SAct 0x3fc0 SErr 0x4040000 action 0xe frozen
Dec 02 10:09:57 nix01 kernel: ata2.00: irq_stat 0x00000040, connection status changed
Dec 02 10:09:57 nix01 kernel: ata2: SError: { CommWake DevExch }
Dec 02 10:09:57 nix01 kernel: ata2.00: failed command: WRITE FPDMA QUEUED
Dec 02 10:09:57 nix01 kernel: ata2.00: cmd 61/08:30:00:00:20/00:00:08:00:00/40 tag 6 ncq 4096 out
                                       res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x10 (ATA bus error)
Dec 02 10:09:57 nix01 kernel: ata2.00: status: { DRDY }
Dec 02 10:09:57 nix01 kernel: ata2.00: failed command: WRITE FPDMA QUEUED
Dec 02 10:09:57 nix01 kernel: ata2.00: cmd 61/08:38:18:00:20/00:00:08:00:00/40 tag 7 ncq 4096 out
                                       res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x10 (ATA bus error)
Dec 02 10:09:57 nix01 kernel: ata2.00: status: { DRDY }
Dec 02 10:09:57 nix01 kernel: ata2.00: failed command: WRITE FPDMA QUEUED
Dec 02 10:09:57 nix01 kernel: ata2.00: cmd 61/08:40:80:00:20/00:00:0c:00:00/40 tag 8 ncq 4096 out
                                       res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x10 (ATA bus error)
Dec 02 10:09:57 nix01 kernel: ata2.00: status: { DRDY }
Dec 02 10:09:57 nix01 kernel: ata2.00: failed command: WRITE FPDMA QUEUED
Dec 02 10:09:57 nix01 kernel: ata2.00: cmd 61/08:48:f0:05:20/00:00:0c:00:00/40 tag 9 ncq 4096 out
                                       res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x10 (ATA bus error)
Dec 02 10:09:57 nix01 kernel: ata2.00: status: { DRDY }
Dec 02 10:09:57 nix01 kernel: ata2.00: failed command: WRITE FPDMA QUEUED
Dec 02 10:09:57 nix01 kernel: ata2.00: cmd 61/08:50:98:06:20/00:00:0c:00:00/40 tag 10 ncq 4096 out
                                       res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x10 (ATA bus error)
Dec 02 10:09:57 nix01 kernel: ata2.00: status: { DRDY }
Dec 02 10:09:57 nix01 kernel: ata2.00: failed command: WRITE FPDMA QUEUED
Dec 02 10:09:57 nix01 kernel: ata2.00: cmd 61/08:58:f8:13:21/00:00:0c:00:00/40 tag 11 ncq 4096 out
                                       res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x10 (ATA bus error)
Dec 02 10:09:57 nix01 kernel: ata2.00: status: { DRDY }
Dec 02 10:09:57 nix01 kernel: ata2.00: failed command: WRITE FPDMA QUEUED
Dec 02 10:09:57 nix01 kernel: ata2.00: cmd 61/08:60:08:01:a0/00:00:19:00:00/40 tag 12 ncq 4096 out
                                       res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x10 (ATA bus error)
Dec 02 10:09:57 nix01 kernel: ata2.00: status: { DRDY }
Dec 02 10:09:57 nix01 kernel: ata2.00: failed command: WRITE FPDMA QUEUED
Dec 02 10:09:57 nix01 kernel: ata2.00: cmd 61/08:68:28:00:a4/00:00:19:00:00/40 tag 13 ncq 4096 out
                                       res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x10 (ATA bus error)
Dec 02 10:09:57 nix01 kernel: ata2.00: status: { DRDY }
Dec 02 10:09:57 nix01 kernel: ata2: hard resetting link
Dec 02 10:09:57 nix01 kernel: ata2: SATA link down (SStatus 1 SControl 300)
Dec 02 10:09:57 nix01 kernel: ata2: hard resetting link
Dec 02 10:09:57 nix01 kernel: ata2: SATA link down (SStatus 1 SControl 300)
Dec 02 10:09:57 nix01 kernel: ata2: limiting SATA link speed to 1.5 Gbps
Dec 02 10:09:57 nix01 kernel: ata2: hard resetting link
Dec 02 10:09:57 nix01 kernel: ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Dec 02 10:09:57 nix01 kernel: ata2.00: supports DRM functions and may not be fully accessible
Dec 02 10:09:57 nix01 kernel: ata2.00: disabling queued TRIM support
Dec 02 10:09:57 nix01 kernel: ata2.00: supports DRM functions and may not be fully accessible
Dec 02 10:09:57 nix01 kernel: ata2.00: disabling queued TRIM support
Dec 02 10:09:57 nix01 kernel: ata2.00: configured for UDMA/133
Dec 02 10:09:57 nix01 kernel: ata2: EH complete

Bizarre. Can't believe this is NixOS-specific. @dhess make sure you have the latest firmware on both drives. Intel have had similar problems in the past.
I actually have a remote machine that "died" with similar disk errors some time during/after 3.18. It may well have been the same issue, will check on it tomorrow. Hopefully there are some earlier generations left.
At this point I would try different combinations of [no]ncq and [no]ncqtrim (latter requires kernel 4.2+) to libata.force.. Note you can pass them from grub/gummiboot rather than rebuild all the time. Also verify that the options are actually picked up (dmesg?).
I'll have a look through the Debian kernel sources and try to find anything remotely related.

I did verify that 'noncq' is reflected in dmesg when specified on the Grub command line.
I do have the latest firmware on the Samsung drive as I just flashed it last night before my most recent attempts to use it. It didn't make any difference.
I've reinstalled Ubuntu 14.04 and will stress-test the machine for a few hours to see if I get any NCQ errors. I believe that version of Ubuntu is running 3.13, so I may upgrade to whichever version of Ubuntu (or maybe Jessie) has a 3.18 or later kernel so the comparison is more relevant.

As a test, I compiled GHC 7.10.2 from source using make -j12 on the Samsung 840 EVO (latest (EXT0DB6Q) firmware), on 3 different versions of Ubuntu: 14.04, 15.04, and 15.10.  I didn't add any kernel command-line options or otherwise try to manually disable NCQ or TRIM support.
GHC built successfully on 14.04 and 15.04 with no SATA/NCQ issues. It's in the process of building on 15.10 as I write this, but I don't feel like more evidence is needed at this point. The system also handled the 14.04 install and 2 full distro upgrades (14.04 -> 15.04, 15.04 -> 15.10) perfectly. I'm now convinced this is not a hardware issue. In NixOS 15.09 with the same hardware, I get NCQ errors as soon as I do anything as simple as editing a file.
Here are the results of dmesg | grep -i ncq on the two most recent Ubuntu versions:
Ubuntu 15.04 (linux-image-generic <IP_ADDRESS>.36):
[    2.347661] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst
[    2.707964] ata2.00: 488397168 sectors, multi 1: LBA48 NCQ (depth 31/32), AA

Ubuntu 15.10 (linux-image-generic <IP_ADDRESS>.21):
[    2.459704] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst
[    2.820158] ata2.00: 488397168 sectors, multi 1: LBA48 NCQ (depth 31/32), AA

No mention of "horkage" in any logs or in dmesg.
I will probably set the SATA mode to IDE in BIOS and use NixOS that way until this is resolved.
I don't suppose there's any easy way to use a Debian/Ubuntu kernel with NixOS, is there? I Googled a bit but came up empty.

For reference, here are the outputs of lsmod and smartctl -a /dev/sda in Ubuntu 15.10:
Module                  Size  Used by
binfmt_misc            20480  1
ipmi_ssif              24576  0
intel_rapl             20480  0
iosf_mbi               16384  1 intel_rapl
x86_pkg_temp_thermal    16384  0
intel_powerclamp       16384  0
coretemp               16384  0
kvm_intel             167936  0
kvm                   512000  1 kvm_intel
crct10dif_pclmul       16384  0
crc32_pclmul           16384  0
aesni_intel           167936  0
aes_x86_64             20480  1 aesni_intel
lrw                    16384  1 aesni_intel
gf128mul               16384  1 lrw
glue_helper            16384  1 aesni_intel
ablk_helper            16384  1 aesni_intel
cryptd                 20480  2 aesni_intel,ablk_helper
sb_edac                28672  0
edac_core              53248  1 sb_edac
input_leds             16384  0
joydev                 20480  0
mei_me                 32768  0
shpchp                 36864  0
mei                    98304  1 mei_me
lpc_ich                24576  0
ioatdma                65536  0
ipmi_si                57344  0
8250_fintek            16384  0
ipmi_msghandler        49152  2 ipmi_ssif,ipmi_si
mac_hid                16384  0
lp                     20480  0
parport                49152  1 lp
autofs4                40960  2
hid_generic            16384  0
usbhid                 49152  0
hid                   118784  2 hid_generic,usbhid
igb                   188416  0
dca                    16384  2 igb,ioatdma
ahci                   36864  4
ptp                    20480  1 igb
libahci                32768  1 ahci
pps_core               20480  1 ptp
i2c_algo_bit           16384  1 igb
wmi                    20480  0

smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.2.0-19-generic] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 840 EVO 250GB
Serial Number:    S1DBNEAD714949K
LU WWN Device Id: 5<PHONE_NUMBER>158f2
Firmware Version: EXT0DB6Q
User Capacity:    250,059,350,016 bytes [250 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Dec  2 22:24:52 2015 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection: 		( 4800) seconds.
Offline data collection
capabilities: 			 (0x53) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 (  80) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   096   096   000    Old_age   Always       -       17820
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       27
177 Wear_Leveling_Count     0x0013   098   098   000    Pre-fail  Always       -       13
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   062   062   000    Old_age   Always       -       38
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   099   099   000    Old_age   Always       -       421
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       21
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -<PHONE_NUMBER>

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Hit the issue again, despite forcing the presence of libata module.
[38621.166800] ata1.00: exception Emask 0x0 SAct 0xe00000 SErr 0x50000 action 0x6 frozen
[38621.166803] ata1: SError: { PHYRdyChg CommWake }
[38621.166805] ata1.00: failed command: WRITE FPDMA QUEUED
[38621.166808] ata1.00: cmd 61/08:a8:c0:78:e5/00:00:12:00:00/40 tag 21 ncq 4096 out
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[38621.166809] ata1.00: status: { DRDY }
[38621.166810] ata1.00: failed command: WRITE FPDMA QUEUED
[38621.166813] ata1.00: cmd 61/08:b0:f8:78:e5/00:00:12:00:00/40 tag 22 ncq 4096 out
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[38621.166814] ata1.00: status: { DRDY }
[38621.166815] ata1.00: failed command: READ FPDMA QUEUED
[38621.166817] ata1.00: cmd 60/08:b8:08:28:60/00:00:02:00:00/40 tag 23 ncq 4096 in
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[38621.166818] ata1.00: status: { DRDY }

It looks like the kernel param is successfully passed during boot. However, it seems like noncqtrim is not being respected? Full output below, but there is a supsicious line:
[    0.289695] ahci 0000:00:1f.2: flags: 64bit ncq pm led clo only pio slum part deso sadm sds apst

Is this expected?
full output:
☭ sudo dmesg -k | grep -i cq
[    0.000000] Command line: BOOT_IMAGE=(hd0,gpt2)//kernels/9xfh1qyj52ibmpgb5lngx5w3248lq7wz-linux-4.3-bzImage systemConfig=/nix/store/m0d0rf0f6malq33dv09azwcs438k4c4s-nixos-15.09.706.45128de init=/nix/store/m0d0rf0f6malq33dv09azwcs438k4c4s-nixos-15.09.706.45128de/init loglevel=4 libata.force=noncqtrim
[    0.000000] Kernel command line: BOOT_IMAGE=(hd0,gpt2)//kernels/9xfh1qyj52ibmpgb5lngx5w3248lq7wz-linux-4.3-bzImage systemConfig=/nix/store/m0d0rf0f6malq33dv09azwcs438k4c4s-nixos-15.09.706.45128de init=/nix/store/m0d0rf0f6malq33dv09azwcs438k4c4s-nixos-15.09.706.45128de/init loglevel=4 libata.force=noncqtrim
[    0.017667] ACPI: 12 ACPI AML tables successfully acquired and loaded
[    0.289695] ahci 0000:00:1f.2: flags: 64bit ncq pm led clo only pio slum part deso sadm sds apst
[    0.596111] ata1.00: FORCE: horkage modified (noncqtrim)
[    0.597066] ata1.00: 500118192 sectors, multi 1: LBA48 NCQ (depth 31/32), AA
[   14.158037] rtsx_pci 0000:02:00.0: rtsx_pci_acquire_irq: pcr->msi_en = 1, pci->irq = 44

I added libata.force=noncqtrim,noncq to my kernelParams and now I'm seeing NCQ disabled:
☭ dmesg -k | grep -i cq
[    0.000000] Command line: BOOT_IMAGE=(hd0,gpt2)//kernels/9xfh1qyj52ibmpgb5lngx5w3248lq7wz-linux-4.3-bzImage systemConfig=/nix/store/lyi1hbkhh6vnq0rg0lw0fcnrwk1ylmps-nixos-15.09.706.45128de init=/nix/store/lyi1hbkhh6vnq0rg0lw0fcnrwk1ylmps-nixos-15.09.706.45128de/init loglevel=4 libata.force=noncqtrim,noncq
[    0.000000] Kernel command line: BOOT_IMAGE=(hd0,gpt2)//kernels/9xfh1qyj52ibmpgb5lngx5w3248lq7wz-linux-4.3-bzImage systemConfig=/nix/store/lyi1hbkhh6vnq0rg0lw0fcnrwk1ylmps-nixos-15.09.706.45128de init=/nix/store/lyi1hbkhh6vnq0rg0lw0fcnrwk1ylmps-nixos-15.09.706.45128de/init loglevel=4 libata.force=noncqtrim,noncq
[    0.017666] ACPI: 12 ACPI AML tables successfully acquired and loaded
[    0.297387] ahci 0000:00:1f.2: flags: 64bit ncq pm led clo only pio slum part deso sadm sds apst
[    0.604122] ata1.00: FORCE: horkage modified (noncqtrim)
[    0.604125] ata1.00: FORCE: horkage modified (noncq)
[    0.604651] ata1.00: 500118192 sectors, multi 1: LBA48 NCQ (not used)
[    9.653970] rtsx_pci 0000:02:00.0: rtsx_pci_acquire_irq: pcr->msi_en = 1, pci->irq = 45

We'll see how long this is stable for.

I've tried the same with AHCI mode turned back on in the BIOS, using a Linux 4.2 kernel:
[    0.000000] Command line: BOOT_IMAGE=(hd0,msdos2)/nix/store/f77jdmsx27a81qkrfvmz7hjh5c83cwkm-linux-4.2.5/bzImage systemConfig=/nix/store/nr83md689m2zlf0byas7y460228p2sy4-nixos-15.09.706.45128de init=/nix/store/nr83md689m2zlf0byas7y460228p2sy4-nixos-15.09.706.45128de/init libata.force=noncqtrim,noncq loglevel=4
[    0.000000] Kernel command line: BOOT_IMAGE=(hd0,msdos2)/nix/store/f77jdmsx27a81qkrfvmz7hjh5c83cwkm-linux-4.2.5/bzImage systemConfig=/nix/store/nr83md689m2zlf0byas7y460228p2sy4-nixos-15.09.706.45128de init=/nix/store/nr83md689m2zlf0byas7y460228p2sy4-nixos-15.09.706.45128de/init libata.force=noncqtrim,noncq loglevel=4
[    0.058802] ACPI: All ACPI Tables successfully acquired
[    0.828099] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst
[    1.145191] ata2.00: FORCE: horkage modified (noncq)
[    1.145215] ata2.00: 488397168 sectors, multi 1: LBA48 NCQ (not used)

Unfortunately, this causes a different error:
[   31.413338] ata2.00: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen
[   31.413368] ata2.00: irq_stat 0x00000040, connection status changed
[   31.413390] ata2: SError: { CommWake DevExch }
[   31.413407] ata2.00: failed command: READ DMA EXT
[   31.413426] ata2.00: cmd 25/00:08:d0:ba:24/00:00:1b:00:00/e0 tag 7 dma 4096 in
                        res 50/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x10 (ATA bus error)
[   31.413475] ata2.00: status: { DRDY }
[   31.413491] ata2: hard resetting link
[   33.618097] ata2: SATA link down (SStatus 1 SControl 300)
[   33.973001] ata2: hard resetting link
[   36.179834] ata2: SATA link down (SStatus 1 SControl 300)
[   36.179842] ata2: limiting SATA link speed to 1.5 Gbps
[   36.474890] ata2: hard resetting link
[   36.779773] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[   36.781716] ata2.00: supports DRM functions and may not be fully accessible
[   36.781889] ata2.00: supports DRM functions and may not be fully accessible
[   36.781892] ata2.00: configured for UDMA/133
[   36.792796] ata2: EH complete

Tried the 4.3 kernel, same result, this time with a WRITE DMA EXT error forcing the controller into UDMA/133 mode.

@dhess how are you forcing these issues to happen so quickly? I only hit this after several hours (and occasionally days)

@aaronlevin For me it reliably happens only a few seconds after login. Just lucky I guess :\

@dhess :(

@dhess to run your system with the ssd in IDE mode, did you have to re-generate hardware-configuration.nix?

No. I just made the BIOS change. Everything is working great now; it's just a shame I had to cripple the SSD performance to get here.

@dhess your last error is closer to what I had on the mentioned remote system (also Supermicro). Didn't get to look at it yet, but can you post lspci -nv? Curious which disk controller you have.

lspci.txt

Can we remove the needs: feedbacck tag on this?

Update: I tried updating my bios, which succeeded (but my version is still the same). however, now I seem to be experiencing the issue in rapid succession.

This is preventing me from being productive at work and I may be forced to abandon this. I really don't want to do this.

@dhess how did you actually put your SSD into IDE mode? I didn't see a setting in the BIOS for that. Is it configured elsewhere? Thanks.

@dhess can you try the latest unstable channel? I'm running 4.2.6 on the (Intel Atom) Supermicro and it's been stable for 30 hours.
@aaronlevin the option is typically called "disable AHCI" or something like that. Since you upgraded the firmware, what does dmidecode | grep -A1 'Vendor: LENOVO' | tail -n 1 produce?

@aaronlevin I believe you're on a Lenovo laptop, whereas I'm on a Supermicro server, so the BIOSs will obviously be quite different; but in my case, the option was located in the "SATA settings" page. For the SATA mode, my BIOS gives a choice of AHCI, RAID, or IDE. I set mine to IDE (default and highest-performance mode is AHCI).
@mbakke I'd prefer to stay on the latest release channel as I want to use this machine to deploy other release-channel hosts using Nixops.
Do you know whether it's possible to run Nixops on the unstable NixOS channel to deploy NixOS hosts on a different (stable) channel? I searched a bit for info about that and concluded that it isn't supported.

@mbakke
☭ sudo dmidecode | grep -A1 'Vendor: LENOVO'| tail -n 1
	Version: N10ET38W (1.17 )

@mbakke is it possible that having allowDiscards in luks was overriding my libata.force=noncq/libata.force=noncqtrim settings? I'm jinxing myself by saying this, but my system has been stable for 1.5 days since (potentially not all correlated):

setting libata.force=noncq
setting libata.force=noncqtrim
force loading libata module to load in initrd
removing allowDiscards from my luks configuration

I spoke to soon. Hit a weird segfault failure (perhaps unrelated), rebooted, and then after 5 hours hit the issue again. UGH.

I just tried the 4.4-rc4 kernel (via nixpkgs master branch) and still encountered the problem.

Some time ago I had similar issues on a Thinkpad T450s. If I remember it correctly, my solution was:
powerManagement.scsiLinkPolicy = "max_performance";

@jgeerds thanks. I just added that and am trying it now. I remember seeing some bug reports with respect to power management, but didn't quite know the right incantation. fingers crossed!

@aaronlevin how is it working? :smiley:

@jgeerds so far, no failures! I need to wait for another day to break the record, but I'm feeling pretty confident about this solution!
@dhess you should try powerManagement.scsiLinkPolicy = "max_performance"; and report back!

I'm thinking about closing this issue as I've had no issues after setting the powerManagement flag. /cc @dhess @mbakke @jgeerds
Thanks everyone for your help! I will re-open if @dhess reports back that the above solution didn't work or if I experience file system failure again.

The docs for that function say max_performance is the default, is that wrong?

@jb55 uh oh, please don't shake the fragile foundation I'm standing on! :)

@jb55 current implementation is "min_power": https://github.com/NixOS/nixpkgs/blob/d7c0612f4bf8f92cc75a29ed6619221ace049644/nixos/modules/config/power-management.nix#L74

Ah k, I was looking at https://github.com/NixOS/nixpkgs/commit/f169f60575bec7c61626af180cae364d321b4bec, I realize now he was just talking about what the kernel defaults to.

Nice catch! powerManagement.scsiLinkPolicy = "max_performance"; seems to have fixed my problems, as well!

@dhess wow! Very glad to hear this. I've submitted a PR to revert to the kernel default. Yay!

If you still have other distros on the same hardware, could you check if echo min_power | tee /sys/class/scsi_host/host*/link_power_management_policy reproduces the failure?

Glad to hear this! :beer:
If I had read this issue a few weeks earlier I could save you a lot of trouble. Sorry for that!


Arrays are not fully supported in patches

The patch Operation "add" does not work for Arrays in the gamestate. Instead, every time a new item gets pushed into it, the Operation for the patch is "replace" and the value is the full Array itself.
I could investigate this a bit further and found the origin of the Problem:
https://github.com/gamestdio/colyseus-unity3d/blob/master/Assets/Colyseus/DeltaListener/Compare.cs#L46
the type of oldVal and newVal in such cases is NOT Dictionary, but Array. So for Arrays the condition is always false and the else-case gets done.
In the JS-Client, this Problem does not exist because typeof [] = 'object'.

I'll try to solve this tomorrow ;)

Hi, I tried to resolve and tested adding new array items to room state, tested also removing, but I am not sure how the replace would work, please have a look at my pull request and evaluate... for my case locally this fix would be probably enough... let's see  #16

when will this be resolved? thx

Fixed on 0.5.0 release! https://github.com/gamestdio/colyseus-unity3d/releases/tag/0.5.0


macos m1pro fatal error: 'rados/librados.h' file not found

ceph-client has been installed
ceph -v
ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
go build -tags quincy
github.com/ceph/go-ceph/rados
/Users/linxianqin/go/pkg/mod/github.com/ceph/go-ceph@v0.22.0/rados/alloc_hint_flags.go:4:11: fatal error: 'rados/librados.h' file not found
#include <rados/librados.h>
^~~~~~~~~~~~~~~~~~
1 error generated.

You'll have to install development packages for RADOS library i.e, librados-devel or librados-dev as mentioned in the Install section of README.

I have installed the ceph library, and I can see the rados header files in the /opt/homebrew/include directory

Yes, macos M1 is installed by default in /opt/homebrew/include and /opt/homebrew/lib, while go-ceph only looks from /usr/local/lib and /usr/local/include.
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/include/rados  /usr/local/include/rados
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/include/rbd  /usr/local/include/rbd
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/include/cephfs  /usr/local/include/cephfs
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/include/radosstriper  /usr/local/include/radosstriper
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/libceph-common.2.dylib /usr/local/lib/libceph-common.2.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/libceph-common.dylib /usr/local/lib/libceph-common.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/libcephfs.2.0.0.dylib /usr/local/lib/libcephfs.2.0.0.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/libcephfs.2.dylib /usr/local/lib/libcephfs.2.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/libcephfs.dylib /usr/local/lib/libcephfs.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/librados.2.0.0.dylib /usr/local/lib/librados.2.0.0.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/librados.2.dylib /usr/local/lib/librados.2.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/librados.dylib /usr/local/lib/librados.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/libradosstriper.1.0.0.dylib /usr/local/lib/libradosstriper.1.0.0.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/libradosstriper.1.dylib /usr/local/lib/libradosstriper.1.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/libradosstriper.dylib /usr/local/lib/libradosstriper.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/librbd.1.17.0.dylib /usr/local/lib/librbd.1.17.0.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/librbd.1.dylib /usr/local/lib/librbd.1.dylib
sudo ln -s /opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/librbd.dylib /usr/local/lib/librbd.dylib

The above problem can be solved by soft connections to /usr/local/lib and /usr/local/include

I tried to use 'export CGO_LDFLAGS="-L/opt/homebrew/lib"' to solve my problem. Although I could build, I would still get an error when running. Show 'ceph - 17.2.5 / build/lib/librados. 2. The dylib' (no to file) '

I see that you set CGO_LDFLAGS to -L/opt/homebrew/lib/. Are you sure that required shared libraries are present under /opt/homebrew/lib/?

Yes, macos M1 series architecture, shared libraries in /opt/homebrew/lib/

Try otool -L [your executable] to find out, if it can find the libraries at runtime.

Similar issue occurred for me. I was able to get my executable to run successfully without symlinks by building with
export CGO_CFLAGS=-I/opt/homebrew/include
export CGO_LDFLAGS="-L/opt/homebrew/lib"

and running with
DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/lib ./ceph_tests
For reference, otool output:
ceph_tests:
        /opt/homebrew/opt/ceph-client/lib/librados.2.dylib (compatibility version 2.0.0, current version 2.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1953.255.0)
        /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

and original runtime error:
dyld[40480]: Library not loaded<EMAIL_ADDRESS>  Referenced from: '/opt/homebrew/Cellar/ceph-client/17.2.5_1/lib/librados.2.0.0.dylib'
  Reason: tried: '/usr/local/lib/libceph-common.2.dylib' (no such file), '/usr/lib/libceph-common.2.dylib' (no such file)
[2]    40480 abort      ./ceph_tests`

@itxiaolin Were you able to make any progress with suggestions from https://github.com/ceph/go-ceph/issues/902#issuecomment-1632472200 and https://github.com/ceph/go-ceph/issues/902#issuecomment-1644182874?


Add an # ignore marker

Not user-facing, private. Added by DTFinder (stopwords), recognized by Checker.

https://github.com/ev-br/scpdt/commit/416b74738aed2169b9ed35ff6c3ac2fd6b4a673d


"Lasso" selection tool for graphs

Being able to select nodes with a lasso tool would be very helpful.

We already something similar, the box selection:
modifier key + mousedown, then drag to extend the box.
Btw we can think to add your suggested feature too.

Thanks! Using shift + mousedown indeed allows selection via a rectangle. This was not immediately apparent to me.
Note that the manual says that the shortcut is CTRL/CMD + click & drag, while for me it is SHIFT.


Network is not resumed

Hi repo owner,
I still have this issue even with firmware upgraded and using your latest code.
The MQTT failed to reconnect after network resumed.
Here is detail:

Firmware version 3.30
Your latest code on branch v1.0-DriveTestFixes
Operate normal at the beginning
Put it into Faraday cage
Take it out
Waited 3 minutes, but no reconnect

...
04:54:17.741: {"Timestamp":1692939257,"CellQuality":{"RSRP":-140,"RSRQ":-33,"RSSI":0,"SNR":-22,"RxQual":-1},"CellInfo":{"LogicalCellID":"0x00000000","PhysicalCellID":"192","EARFCN":9410,"PLMN":50501,"Operator":"Telstra Mobile"}}
04:54:35.712: Network status unknown.
04:54:35.876: Network status unknown.
04:54:51.916: Network connection is down...
04:55:22.102: Network connection is down...
04:55:52.287: Network connection is down...
04:56:22.637: Network connection is down...
04:56:52.823: Network connection is down...
04:57:23.175: Network connection is down...
04:57:53.361: Network connection is down...
04:58:23.715: Network connection is down...
04:58:53.903: Network connection is down...
04:59:24.260: Network connection is down...
04:59:54.448: Network connection is down...
05:00:24.806: Network connection is down...
05:00:54.993: Network connection is down...
05:01:25.355: Network connection is down...
05:01:55.543: Network connection is down...
05:02:25.906: Network connection is down...
05:02:56.092: Network connection is down...
05:03:26.459: Network connection is down...
05:03:56.645: Network connection is down...
05:04:27.009: Network connection is down...
05:04:57.196: Network connection is down...
05:05:27.564: Network connection is down...

// Out from Faraday cage

05:05:57.750: Network connection is down...
05:06:28.120: Network connection is down...
05:06:58.308: Network connection is down...
05:07:28.680: Network connection is down...
05:07:58.868: Network connection is down...
05:08:29.244: Network connection is down...
05:08:59.431: Network connection is down...
05:09:29.807: Network connection is down...
05:09:34.232: Exit button pressed, closing down... Please wait for the RED LED to go out...
05:09:49.321: Waiting for app tasks to stop... This can take sometime if waiting for AT commands to timeout...
05:09:49.433: All tasks have now finished...
05:09:49.539: Stop Registration task requested...
05:09:49.545: Waiting for Registration task to stop...
05:09:49.551: De-registering from the network...
05:09:50.218: Network status unknown.
05:09:50.223: Network status unknown.
05:09:50.333: Deregistered from cellular network
05:09:51.550: Application Finished.

Cheers,
Paul

Depending on the MNO Profile you have set will determine the number of bands that it will scan over after network out-of-coverage event.  The SARA-R5 module should reconnect to the network after a period of time, once it has re-started it's cell and band scanning.
This issue is to do with the R5 cellular module, not the application, as the application only reports on the +CEREG status and has no input on how the R5 operates when out-of-coverage.
Can you leave the module turned on for longer after you remove it from the cage?

I don't see any issue with the network resumption and although it can take a longer time than expected, it does recover.
This is all down to the back-off timers of the cell and band scanning when the module goes out of coverage. The back off is to save on power consumption.


Missing Required "EnableTeamTier_Internal" registry Key

This document is missing the required "EnableTeamTier_Internal" registry key. It used to be referenced in a ZIP file in the requirements section, but it has been removed.
Example:
Opt-in to the Insiders Program for OneDrive

Currently On-prem sync functionality is only available to users that are part of the OneDrive “Insiders” program. You can opt-in to the program using the following registry key addition:

reg.exe add HKCU\SOFTWARE\Microsoft\OneDrive /v EnableTeamTier_Internal /t REG_DWORD /d 1 /f

After adding the registry key, new settings will be added to OneDrive. These setting are stored in a file name “PreSignInSettingsConfig.json”.
Note: This file is automatically refreshed every 24 hours, but it can be manually refreshed by deleting the setting files and re-starting the OneDrive client.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8f0d9527-a78b-d18b-d218-881fac11423d
Version Independent ID: 3ad8fc7b-4869-dbdc-802b-a804f65d0e39
Content: Configure syncing with the new OneDrive sync client for Sharepoint 2019
Content Source: SharePoint/SharePointServer/install/new-onedrive-sync-client.md
Product: sharepoint-server-itpro
GitHub Login: @kaarins
Microsoft Alias: kaarins

@AndreaBarr assign to me, please.
@mikelee-msft any idea about how long this will be valid (i.e. when this will go into the production ring of release)?

Just had a quick conversation with the PG contact for NGSC. It looks like the newest build no longer requires this flag. I guess we close this feedback since it's not needed.


RbVmomi deserialization fails on managed object type ContentLibrary

Describe the bug
Running a simple WaitForUpdates event catcher (watching latestPage from the eventManager) [ref] and observed an exception when a scheduled "Sync Library" task ran.  When we caught this event an exception was raised by deserialization because it couldn't find the ContentLibrary type.  Further investigation shows that there was a task with an entity of ManagedObjectReference:ContentLibrary clib-2895
To Reproduce
Steps to reproduce the behavior:

Synchronize a Content Library
vim.serviceContent.taskManager.recentTask.first.info
:boom:

>> vim.serviceContent.taskManager.recentTask.first.info
Traceback (most recent call last):
       16: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:106:in `traverse_data'
       15: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:106:in `each'
       14: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:119:in `block in traverse_data'
       13: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:90:in `deserialize'
       12: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:106:in `traverse_data'
       11: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:106:in `each'
       10: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:124:in `block in traverse_data'
        9: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:90:in `deserialize'
        8: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:106:in `traverse_data'
        7: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:106:in `each'
        6: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:124:in `block in traverse_data'
        5: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/deserialization.rb:87:in `deserialize'
        4: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/type_loader.rb:65:in `get'
        3: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/type_loader.rb:65:in `synchronize'
        2: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/type_loader.rb:67:in `block in get'
        1: from /home/grare/adam/.gem/gems/rbvmomi-3.0.0/lib/rbvmomi/type_loader.rb:105:in `make_type'
RuntimeError (unknown VMODL type ContentLibrary)

Expected behavior
Given that ContentLibrary isn't a registered ManagedEntity in the Managed Object Types I expect this is one of those special types that has to be merged into the vmodl.db
Screenshots
The task in the vSphere client:

TaskInfo from the MOB:

Platform:

OS: Debian Bullseye
RbVmomi Version: 2.4.1 and 3.0.0
Ruby Version ruby 2.7.2p137 (2020-10-01 revision 5445e04352

Sorry, I haven't gotten to this yet. The content library APIs are not available from SOAP. That's why it isn't registered as a ManagedEntity. I'll see what I can do to have it go around this.

No problem @jrgarcia  If you know how you'd like to handle it I can take a crack at fixing it.
I could see adding a "stub" definition to the vmodl.wsdl or more generally changing the deserializer to create an object of the base class type for that property if it can't find the class in the vmodl.
Differentiating between an actual error and a case like this would be tricky though, but I don't know how many other cases like this there might be (where an internal definition "leaks" out over the public API).
This is mostly just annoying for us because it causes our WaitForUpdatesEx loops to raise out and we have to start from version="" again.

Hey @jrgarcia!  I see you're no longer with VMware, congrats on the new job!  Best of luck.
Do you know if there is there a new maintainer of this gem?

Thanks, @agrare! Unfortunately, I do not know who will be working on this going forward. I'll see if I can find an answer for you, but as I have no access to anything internal to VMware anymore, I'm not sure I'll be able to get an answer too quickly.


Cannot build master branch

Godot version:
Master
OS/device including version:
Ubuntu 16.04
Issue description:
When building master, I get errors while compiling editor/plugins/visual_shader_editor_plugin.cpp
On all the lines with Shader::Mode I get the error: 'Shader::Mode' is not a class or namespace
It looks like travis-ci is building fine. But im consistently getting errors.
@Chaosus Do you have any idea what might cause this? Could it be caused by https://github.com/godotengine/godot/pull/28571?

If it helps.  As of c50ab02 on Windows 10 with a 64-bit VS2017 build, I didn't encounter it.

Thanks! I was just able to successfully build on my windows machine too. I can't figure out why I would be having problems.

That's strange, it's still happens on Ubuntu ?

Ah.. probably it's not okay to call a full "Shader::Mode::" - need to call just a "Shader::" - will fix it now

That PR worked. Thanks!


Rejecting a CONNECT request with an html message?

I am trying to Reject CONNECT requests via
proxy.OnRequest().HandleConnectFunc(func(host string, ctx *goproxy.ProxyCtx) (*goproxy.ConnectAction, string) {
            ctx.Resp = goproxy.NewResponse(ctx.Req,
                    goproxy.ContentTypeHtml, http.StatusForbidden,
                    "

Address is blocked.
")
            return goproxy.RejectConnect, host
}

But this doesn't seem to work, browser shows the connection problem page instead of the html.

+1

@ismail took a look at abourget/goproxy ? you can short-circuit any handlers with a ctx.NewResponse() .. including the CONNECT handlers..

@osiloke read previous comment too


Update footer.js

Summary
Briefly describe the purpose of this PR.
Fixed bugs + Add Discord feature in footer
Description
Explain in detail what this PR is all about. This should include the problem you're solving, the approach taken, and any technical details that reviewers need to know.
Added Links of github in footer + Added discord in our social media section with hover effect icon and link.
Images
Include any relevant images or diagrams that can help reviewers visualize the changes, if applicable

Issue(s) Addressed
Enter the issue number of the bug(s) that this PR fixes

Template should be strictly Closes <issue_number>
Example: Closes #1390

Prerequisites

[X] Have you followed all the CONTRIBUTING GUIDELINES?

@Its-Aman-Yadav sir please review prs since the deadline is 7pm today so I request you to please review as soon as possible.  Thanks

@Its-Aman-Yadav @Kalivarapubindusree please review before 7pm

@Its-Aman-Yadav conflicts resolved


Use Turbopack or SWC for transpiling electron app + cleanup config files

https://github.com/crusher-dev/crusher/tree/master/packages/electron-app/webpack di contains config for building crusher electron app
We build for renderer process and electron-main process.
Tasks
:- use swc-loader instead of ts-loader. File for example(https://github.com/crusher-dev/crusher/blob/master/packages/electron-app/webpack/webpack.renderer.ts#L32)
:- Cleanup webpack config files

@himanshu-dixit can I work on it?

@arpitBhalla :- assigned it to you


Updating readme from branch example

Updates
I updated the readme file Please review and accept

Thoughts
I probably shouldn't, but okay. 👍


Passphrase error

After created, I can change the passphrase, but I can't use it to unlock my wallet or to send coins.
Windows 10.

I have noticed I get a Incorrect Passphrase error too, although it does in fact unlock my wallet. It is sending the Incorrect Passphrase message for both true and false.


Community Board UI Update

Overview
This update adds the UI for the community board, including the trending screen, flyers screen, and reads screen.
Changes Made

Whole app restructured, Trending is now home screen, Reads page contains articles, magazines, and publications. Flyers screen shows community board.
Articles now have shimmer loading effect in new screens.
Networking for Flyers not yet implemented, but everything else works.

Next Steps

Implement flyers networking.
Implement cover images networking for magazines.

Screenshots & Videos

Great work!!!
A few issues to point out: in the flyer tab, in the upcoming portion, the location is slightly cut off. Please fix that.  Other than that wonderful work


fix: update to eye v9.7.10

fix: update to eye v9.7.10

:tada: This PR is included in version 12.5.7 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release
v12.5.7

Your semantic-release bot :package::rocket:


[DO NOT MERGE] Debugging network smoke test flake

/hold

/retest

CI failed to clone src image
/retest

/retest

/retest

/retest

/retest

/retest

Closing the PR as there are already plenty of instances of the failure in CI with available logs.


Tutorial errors out

The tutorial to train a model from scratch errors out and complains about a wrong keyword argument:
DataLoader(::Any, ::Any; collate, buffered, partial, useprimary) at ~\.julia\packages\DataLoaders\uGlPg\src\DataLoaders.jl:54 got unsupported keyword argument "validbsfactor"
DataLoader(::Any) at ~\.julia\packages\DataLoaders\uGlPg\src\DataLoaders.jl:54 got unsupported keyword argument "validbsfactor"

I followed the stack trace to DLPipelines.jl and DataLoaders.jl and can't find validbsfactor anywhere. What does it do?

PR in preparation to fix

Solved by the release of DLPipelines.jl v0.2.0


baseUrl configuration

General Question
Form
Put an [x] if you meet the condition, else leave [ ].

[X] I've searched the Issues
[X] I've read the basic concepts
[X] I'm using the latest version

Question
Hi guys! Thank you for the hard work on building this library. I wonder if there's any way of configuring the instance so that I can change baseUrl from https://i.instagram.com to one of my choice. The reason is that I created an instagram mock server (with the help of Wiremock) that I use for testing. Being able to use it would help me massively to test my cases without hitting the real instagram.

Change the code in request.js
You can configure base URL.


App scaling for large displays

📝 App scaling for large displays
What is the expected behavior of the proposed feature?  What is the scenario this would be used?
---  Would be great to have several zones on my tv without the application menus taking up most of the zone (just to have them the same size as on my laptop screen).  For example:

Excel, for example shows 29 rows on my laptop screen compared to 13 with a similar sized zone on my tv:

If you'd like to see this feature implemented, add a 👍 reaction to this post.

Excel, for example shows 29 rows on my laptop screen compared to 13 with a similar sized zone on my tv:

Sorry if I'm not understanding, but both images/screens are 1920x1080 and in the first image Excel has only 1/2 x 1/2 of the screen. You want it to scale down, so that it shows the same relative size? That might be very difficult for the monitor, something tells me those pixels are not going to like that.

Hi Jay,
Yes that is correct, although I‘m not sure about the difficulty of it.
In my head it would be like treating the zone as if it were another display.

I added the suggestion to https://github.com/microsoft/PowerToys/issues/279#issuecomment-791048169 since this is not doable in FZ. It requires virtual monitors supported by the system.


[FEAT] Detect when a project does not have plugin added and notify user

Describe the feature
When our plugin is added, most DOM elements should have the data-onlook-id attribute. We can scan the DOM tree for this attribute or just pick it up as we parse the DOM tree. If none is found, we should let the user know and prompt them to add the plugin.
@drfarrell thoughts on how this should look on the user end? I would think an error indicator on the code side or on the browser page should work.

I like both approaches – something in the editor and perhaps a persistent toast in the corner OR as a modal on the front page.
If it's disruptive as a large modal though, it'll need some good documentation though, so definitely something to consider.

I think a persistent icon on the code-related modules would do


Do not bother user with rebuilds that don’t change the result

I believe this is not a precise duplicate of any existing issues, but I might be wrong, in that case, sorry.
My nix-shell involes some evaluation-time computation that is easily triggered (in fact, it reads .git/index). Lorri detects that it uses that file (good), watches that file for changes (good), re-starts a build when that files changed (good), and when the build is finished, even thought the resulting derivation is identicaly, reloads my direnv (bad, because very noisy).
Maybe this is naive, but couldn’t lorri notice that the shell derivation is unchanged in most cases, and not do anything?

Maybe this is naive, but couldn’t lorri notice that the shell derivation is unchanged in most cases, and not do anything?

It should, by just comparing the store path of the new gc_root with the old one, and not replacing it if it doesn’t change.

The root_result call here, should be a fairly easy change:
https://github.com/target/lorri/blob/200f4a2b5b1bbf9a2dd8c9b51c2cb907c7aebab3/src/build_loop.rs#L187-L195


FreeTSA Missing

Hi,
Not sure when it happened, but FreeTSA is now missing in the official release.
I see it's added back like 20 days ago here:
https://github.com/bitbeans/SimpleDnsCrypt/commit/5a6d7cfc1d32a69180900cbbafa0cb91b8ba7d24
New release to fix this?
Edit: Guess because no longer listed @ https://github.com/jedisct1/dnscrypt-proxy/blob/master/dnscrypt-resolvers.csv
Edit 2: I see now: https://github.com/jedisct1/dnscrypt-proxy/commit/302860b2cbc5441bf749ab366c880307037b019f#diff-bc41fce4beb60b51d3b36a920e4b3a9a
But seems it's back up and working fine again.
Thanks,
Will

@war59312 dnscrypt-resolvers.csv is not maintained in this project :)


Update app.js

Updating the backend port to 80.

@rishirsinha,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Update Grid Documentation for 0.12

The existing grid has issues that cannot be solved as they are, so we wish to provide an updated version of the grid and align the migration strategy for easy adoption.

[x] Pull in latest Bootstrap 4 grid
[x] Prefix classes with clr- naming convention
[x] Documentation updates

Merged into website.


Release app

This pr updates the README.md with the link to the Releases page, where the app can now be downloaded.

Actually I will remove the apk from the pr, it is not needed I think and it is better to not push it on github. It will still be available under releases though


issue with the display of multiple line annotations

The quote block seems to be only applying to the first line, which appears quite strange.

We are working on creating highlight quote in markdown format in Omnivore to retain some of the HTML format including multi-line paragraphs

I think this is resolved with the new markdown highlights now. You would have to create a new highlight though to see the changes, it wont effect old highlights.

@jacksonh I think this probably will be partially solved because there will still be issues with blockquote of multiple paragraphs: https://www.markdownguide.org/basic-syntax/#blockquotes-with-multiple-paragraphs
I have an idea to solve this but not finished yet

Hey @yifeitao, we have fixed this in the latest release v1.12.4
Kindly upgrade your plugin and let me know if the issue persists.
Thanks!


Provide a replacement for legacy logging libraries

Reliability Kit should host a replacement for the legacy Customer Products logging libraries:

n-logger
n-mask-logger
n-serverless-logger

What problem does this feature solve?
We have three logging libraries when really we only need one. The existing libraries:

are large because they still contain legacy code for logging to Splunk without log drains
are not well-understood by our engineers
use old unmaintained versions of Winston which will eventually become security issues

Ideal solution
We should provide a single logging solution built into Reliability Kit which caters for the needs of all three existing logging libraries: working on Heroku, working on AWS Lambda, and being able to mask sensitive data in the logs. This should be achievable as the APIs for these libraries are fairly similar.
We should take the approach of the new logging library being a drop-in replacement for the existing libraries where possible. We want the migration to be very easy. We should mark some of the existing methods as deprecated in order to design a more consistent logging API.
We should only provide logs in JSON format, as that's what we now expect in Splunk. We should not solve the problem of prettifying logs for local development as this generates a lot of additional code, and there are good third-party libraries for JSON log prettification (e.g. pino-pretty)
We should use Pino as a logger under the hood. It's very fast and logs asynchronously so it should have a positive performance impact in our codebases.
Alternatives

We could do nothing, but we end up maintaining the three logging libraries ad-infinitum. This is a maintenance overhead if we decide to do any work in them. If we do not do any work in them then the security risk increases.

We could fix up the existing logging libraries and try to consolidate functionality into one. I think we're reaching the point where we'd rewrite so much of the code that it'd be more efficient and timely to replace them with something new.

Reopening as we still need a migration path from n-mask-logger.


Change parameter mapping, add missing parameters, add multiprocessing to parsing

I updated the available data to include the missing parameters, mentioned by @meteoDaniel . Also I added the missing time resolution for subdaily . Furthermore when testing the 1_minute resolution, I recognized slow processing when parsing the data. Therefor I added a parallel option to leverage the cores of the computer as well.
As the parallel option creates overhead, it is now checked in collect_dwd_data if the time resolutions are 1_minute or 10_minutes. Within those resolutions there are multiple files for a station. For other resolutions (hourly and above) there's only one single file thus it doens't make sense to use multiprocessing.
Also looking at the solar parameter, I changed its period type to recent, which is the official period according to DWD documents. Also for now I removed the fuzzy parser for DWDStationRequest, as it does not offer  a solid base and should rather be replaced by string calculations based on other libraries, that are able to match different kinds of strings and provide a probability of choice.
Check it out and leave your thoughts please!

Thanks for improving the code base on various angles. Looks good to me.


Discussion: unordered  / ordered insertMany for tables

currently if there is a typing issue etc it looks like we do not run the whole batch, but this is a little different to collections.
Discussion to be had

closing, duplicate with https://github.com/stargate/data-api/issues/1672


[Feature Request] Expand / Collapse all rows programmatically

Existing Component
Yes
Component Name
Table
Description
I want to expand / collapse all rows programmatically. How to do that? Need to loop all rows? There's a columns collection. But I don't find rows collection. Please advise. Thanks.

The link is not working in IE. But I think it's IE forEach problem.


fix calendar

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

mixed-reality-docs/sf-academy-events.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


errors in bam

Running $ picard ValidateSamFile INPUT=xxxxx-ready.bam (bam file produced by bcbio-nextgen) gives the error listed below. Is this due to bam files being merged? Any potential problems from this, reason to worry?
Adam Ewing, who makes BamSurgeon pointed this out.
[Fri Feb 05 10:35:56 CET 2016] picard.sam.ValidateSamFile INPUT=./calling/final/15-8571/15-8571-ready.bam    MODE=VERBOSE MAX_OUTPUT=100 IGNORE_WARNINGS=false VALIDATE_INDEX=true IS_BISULFITE_SEQUENCED=false MAX_OPEN_TEMP_FILES=8000 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json
[Fri Feb 05 10:35:56 CET 2016] Executing as<EMAIL_ADDRESS>on Linux 3.10.0-327.4.4.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_65-b17; Picard version: 1.141(8ece590411350163e7689e9e77aab8efcb622170_1447695087) IntelDeflater
ERROR: Record 7979, Read name M00568:145:AM0EH:1:1118:28619:18261, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 7980, Read name M00568:145:AM0EH:1:1118:28619:18261, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 7980, Read name M00568:145:AM0EH:1:1118:28619:18261, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 7979, Read name M00568:145:AM0EH:1:1118:28619:18261, Both mates are marked as first of pair
ERROR: Record 74063, Read name M00568:145:AM0EH:1:1105:25763:21317, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 74065, Read name M00568:145:AM0EH:1:1105:25763:21317, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 74065, Read name M00568:145:AM0EH:1:1105:25763:21317, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 74063, Read name M00568:145:AM0EH:1:1105:25763:21317, Both mates are marked as first of pair
ERROR: Record 97668, Read name M00568:145:AM0EH:1:1107:11206:14110, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 97724, Read name M00568:145:AM0EH:1:1107:11206:14110, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 97724, Read name M00568:145:AM0EH:1:1107:11206:14110, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 97668, Read name M00568:145:AM0EH:1:1107:11206:14110, Both mates are marked as first of pair
ERROR: Record 117910, Read name M00568:145:AM0EH:1:1109:28056:8208, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 117911, Read name M00568:145:AM0EH:1:1109:28056:8208, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 117911, Read name M00568:145:AM0EH:1:1109:28056:8208, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 117910, Read name M00568:145:AM0EH:1:1109:28056:8208, Both mates are marked as first of pair
ERROR: Record 118098, Read name M00568:145:AM0EH:1:1109:11232:6691, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 118099, Read name M00568:145:AM0EH:1:1109:11232:6691, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 118099, Read name M00568:145:AM0EH:1:1109:11232:6691, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 118098, Read name M00568:145:AM0EH:1:1109:11232:6691, Both mates are marked as first of pair
ERROR: Record 138749, Read name M00568:145:AM0EH:1:1113:10503:17115, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 138750, Read name M00568:145:AM0EH:1:1113:10503:17115, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 138750, Read name M00568:145:AM0EH:1:1113:10503:17115, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 138749, Read name M00568:145:AM0EH:1:1113:10503:17115, Both mates are marked as first of pair
ERROR: Record 223524, Read name M00568:145:AM0EH:1:2103:26642:5302, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 223662, Read name M00568:145:AM0EH:1:2103:26642:5302, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 223662, Read name M00568:145:AM0EH:1:2103:26642:5302, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 223524, Read name M00568:145:AM0EH:1:2103:26642:5302, Both mates are marked as first of pair
ERROR: Record 323929, Read name M00568:145:AM0EH:1:2102:24925:19143, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 323930, Read name M00568:145:AM0EH:1:2102:24925:19143, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 323930, Read name M00568:145:AM0EH:1:2102:24925:19143, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 323929, Read name M00568:145:AM0EH:1:2102:24925:19143, Both mates are marked as first of pair
ERROR: Record 444630, Read name M00568:145:AM0EH:1:1110:27417:20278, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 444823, Read name M00568:145:AM0EH:1:1110:27417:20278, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 444823, Read name M00568:145:AM0EH:1:1110:27417:20278, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 444630, Read name M00568:145:AM0EH:1:1110:27417:20278, Both mates are marked as first of pair
ERROR: Record 569855, Read name M00568:145:AM0EH:1:2101:8957:10090, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 569856, Read name M00568:145:AM0EH:1:2101:8957:10090, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 569856, Read name M00568:145:AM0EH:1:2101:8957:10090, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 569855, Read name M00568:145:AM0EH:1:2101:8957:10090, Both mates are marked as first of pair
ERROR: Record 652702, Read name M00568:145:AM0EH:1:2110:24707:14957, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 652704, Read name M00568:145:AM0EH:1:2110:24707:14957, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 652704, Read name M00568:145:AM0EH:1:2110:24707:14957, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 652702, Read name M00568:145:AM0EH:1:2110:24707:14957, Both mates are marked as first of pair
ERROR: Record 656602, Read name M00568:145:AM0EH:1:2113:29297:16539, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 656603, Read name M00568:145:AM0EH:1:2113:29297:16539, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 656603, Read name M00568:145:AM0EH:1:2113:29297:16539, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 656602, Read name M00568:145:AM0EH:1:2113:29297:16539, Both mates are marked as first of pair
ERROR: Record 805041, Read name M00568:145:AM0EH:1:1119:15851:18460, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 805042, Read name M00568:145:AM0EH:1:1119:15851:18460, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 805042, Read name M00568:145:AM0EH:1:1119:15851:18460, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 805041, Read name M00568:145:AM0EH:1:1119:15851:18460, Both mates are marked as first of pair
ERROR: Record 819781, Read name M00568:145:AM0EH:1:1113:18409:4146, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819784, Read name M00568:145:AM0EH:1:1113:18409:4146, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819784, Read name M00568:145:AM0EH:1:1113:18409:4146, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819781, Read name M00568:145:AM0EH:1:1113:18409:4146, Both mates are marked as first of pair
ERROR: Record 819796, Read name M00568:145:AM0EH:1:2110:7399:17403, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819797, Read name M00568:145:AM0EH:1:2110:7399:17403, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819797, Read name M00568:145:AM0EH:1:2110:7399:17403, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819796, Read name M00568:145:AM0EH:1:2110:7399:17403, Both mates are marked as first of pair
ERROR: Record 819881, Read name M00568:145:AM0EH:1:1103:3753:6208, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819882, Read name M00568:145:AM0EH:1:1103:3753:6208, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819882, Read name M00568:145:AM0EH:1:1103:3753:6208, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819881, Read name M00568:145:AM0EH:1:1103:3753:6208, Both mates are marked as first of pair
ERROR: Record 819887, Read name M00568:145:AM0EH:1:2117:20900:22386, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819893, Read name M00568:145:AM0EH:1:2117:20900:22386, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819893, Read name M00568:145:AM0EH:1:2117:20900:22386, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819887, Read name M00568:145:AM0EH:1:2117:20900:22386, Both mates are marked as first of pair
ERROR: Record 819905, Read name M00568:145:AM0EH:1:2102:8672:13686, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819935, Read name M00568:145:AM0EH:1:2102:8672:13686, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819935, Read name M00568:145:AM0EH:1:2102:8672:13686, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819905, Read name M00568:145:AM0EH:1:2102:8672:13686, Both mates are marked as first of pair
ERROR: Record 819907, Read name M00568:145:AM0EH:1:2102:8672:13807, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819937, Read name M00568:145:AM0EH:1:2102:8672:13807, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819937, Read name M00568:145:AM0EH:1:2102:8672:13807, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819907, Read name M00568:145:AM0EH:1:2102:8672:13807, Both mates are marked as first of pair
ERROR: Record 819917, Read name M00568:145:AM0EH:1:2113:21600:14709, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819970, Read name M00568:145:AM0EH:1:2113:21600:14709, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819970, Read name M00568:145:AM0EH:1:2113:21600:14709, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819917, Read name M00568:145:AM0EH:1:2113:21600:14709, Both mates are marked as first of pair
ERROR: Record 819919, Read name M00568:145:AM0EH:1:2113:21746:15141, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819981, Read name M00568:145:AM0EH:1:2113:21746:15141, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819981, Read name M00568:145:AM0EH:1:2113:21746:15141, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819919, Read name M00568:145:AM0EH:1:2113:21746:15141, Both mates are marked as first of pair
ERROR: Record 819921, Read name M00568:145:AM0EH:1:2113:21854:15193, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819999, Read name M00568:145:AM0EH:1:2113:21854:15193, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 819999, Read name M00568:145:AM0EH:1:2113:21854:15193, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819921, Read name M00568:145:AM0EH:1:2113:21854:15193, Both mates are marked as first of pair
ERROR: Record 819915, Read name M00568:145:AM0EH:1:2113:21860:14704, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 820004, Read name M00568:145:AM0EH:1:2113:21860:14704, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 820004, Read name M00568:145:AM0EH:1:2113:21860:14704, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819915, Read name M00568:145:AM0EH:1:2113:21860:14704, Both mates are marked as first of pair
ERROR: Record 819909, Read name M00568:145:AM0EH:1:2113:21988:14500, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 820033, Read name M00568:145:AM0EH:1:2113:21988:14500, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 820033, Read name M00568:145:AM0EH:1:2113:21988:14500, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819909, Read name M00568:145:AM0EH:1:2113:21988:14500, Both mates are marked as first of pair
ERROR: Record 819913, Read name M00568:145:AM0EH:1:2113:21992:14684, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 820035, Read name M00568:145:AM0EH:1:2113:21992:14684, Mate unmapped flag does not match read unmapped flag of mate
ERROR: Record 820035, Read name M00568:145:AM0EH:1:2113:21992:14684, Mate CIGAR string does not match CIGAR string of mate
ERROR: Record 819913, Read name M00568:145:AM0EH:1:2113:21992:14684, Both mates are marked as first of pair
Maximum output of [100] errors reached.
[Fri Feb 05 10:36:02 CET 2016] picard.sam.ValidateSamFile done. Elapsed time: 0.10 minutes.
Runtime.totalMemory()=669515776
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp

Pär;
Thanks for checking in on BAM correctness. Are you noticing problems processing the BAM downstream? I've always found ValidateSamFile to be too stringent so don't work with it regularly. With test datasets I can replicate getting CIGAR string reports when using bwa on pretty vanilla inputs but not issues with both mates being marked as first.
If it's worth digging into more, I'd likely need to know more about your setup to try and replicate: what type of inputs do you have (standard fastq files, previously aligned BAMs, how long are the reads), what does your configuration look like (are you doing split alignments, recalibration/realignment?), what genome are you aligning against (hg19 versus hg38). Happy to try and help more if this is causing problems.

As far as I know there are no problems, I just though I should inform you if the results would be a reason for concern. But of course you already knew about this..

Pär;
Thanks much. If you run into anything or identify issues happy to try and debug more. Closing for now but re-open if anything comes up.


Backport a couple things from apache/master

Just a couple things that have been fixed upstream which will make ReflectData better to use:

Use Avro 'bytes' type for byte[]
Allow @Union(Class[]) for fields

Hey will its Travis who worked with you previously 2020 happenings. Not sure why I'm subscribed to this email group on my personal email. I'm going to unsubsidized but wanted to let you know in case anyone else might still be attached who shouldn't be any more.  Hope all is well, later!Sent from my T-Mobile 4G LTE Device.
-------- Original message --------From: William Lichtenberger @.> Date: 6/27/22  4:02 PM  (GMT-06:00) To: sporting-innovations/avro @.> Cc: Subscribed @.***> Subject: Re: [sporting-innovations/avro] Backport a couple things from apache/master (#2)
It won't let me re-open this and assign it back to you.  Otherwise I would.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>


Add a Sample for Other Xenforo-based Sites (e.g. AlternateHistory.com)

Could you please add a sample/example for AH.com?

Basically, AH.com will now work. I haven't given a generic example of how to do it via the JSON config, though it wouldn't be too hard, only because it's simpler to add to the registered xenforo sites like I did in that commit.

Is it possible to download full threads? I ask because it's hard to do that in other ways (for example fanficfare forbids it for fear people will misuse it). There are story-only threads in AH that have no threadmarks and can only be downloaded in bulk.

I'm getting 403 error when it tries to access page 2 of a thread. When I try an arbitrary site with page 1 only (removing the next page selector) the epub gets created with page 1 content. (Missing images but I presume I'm missing some configuration.)
I was expecting this to work (file saved in example folder, reluctant-fuhrer.json):
{     "url": "https://www.alternatehistory.com/forum/threads/an-extremely-reluctant-fuhrer-story-only-thread.388640/",     "title": "An Extremely Reluctant Fuhrer",     "author": "johnboy",     "content_selector": "li.message article blockquote.messageText",     "filter_selector": ".sharedaddy, .wpcnt, style",     "next_selector": "link[rel=next]" }
I have the logins saved in leech.json:
{     "logins": {         "AlternateHistory": ["****", "******"]     } }


Review: Landing and About Us

Review Branch
review-landing-and-about-us
Files to review

Landing.jsx
AboutUs.jsx

Checklists

Architecture
Design
JS
React

Due date
1/29/2024 @ 10:30 am
For more information
The review process is documented at: http://courses.ics.hawaii.edu/ics414s21/morea/review/reading-idpm-review.html

Kenji Sanehira Review Notes:
Landing.jsx
Design-01: Some refactoring can be done to make the 3 columns more readable.
Design-08: Similar to design-01, the three columns can be condensed.
AboutUs.jsx
N/A: For the current contents of the page looks fine for now, just need to add appropriate images.

AboutUs.jsx:
Nothing wrong, just have to add actual images in place of the place holders.
Landing.jsx:
Looks alright, though code could be a bit more readable. However, not the end of the world.

AboutUs.jsx:
Good to go.
Landing.jsx
Also good to go.

AboutUs.jsx:
Looks good. Just be sure to take out the comments that are labeled To Do. Maybe we can add those in the related issue.
Formatting is good. Correct use of arrow function.
Landing.jsx:
Looks good. Maybe break the paragraphs into new lines.

Final Review Notes:
Landing.jsx:
Design-01: Refactor cards, use CSS for image.

Code could be more readable.
Break paragraphs into new lines.
AboutUs.jsx:
Design-01: Use CSS for image.
Add Image instead of placeholders.
Change TODO comments into issues.

These will be refactored into a new issue in the M2 project board.


OSX brew consideration

homebrew wants a terminal-based way to test the app, perhaps it could
have a --help flag for the binary that prints version number to the
terminal and then exits?

FWIW a copy of the current brew template here for posterity.
class TarsnapGui < Formula
  desc "A GUI for Tarsnap, making online backups for the truly lazy"
  homepage "https://www.tarsnap.com/"
  url "https://github.com/Tarsnap/tarsnap-gui/archive/v0.5.tar.gz"
  sha256 "33009d710ff736900af4f32b063b3635c5b002005c6b7184cb00d5132d5198f9"
  head "https://github.com/Tarsnap/tarsnap-gui.git"

  depends_on "qt5"
  depends_on "tarsnap"

  def install
    system "qmake"
    system "make"
    prefix.install 'Tarsnap.app'
  end

  test do
    system "true"
  end
end

I tried to quickly find some info on the terminal test you mentioned, but couldn't find anything relevant quickly. Are you talking about a local system test or the auto build bot?

test do
system "true"
end

What does this do?
I also found:

Stuff that builds a .app

Don’t make your formula build an .app (native OS X Application); we don’t want those things in Homebrew. Make it build a command line tool or a library. However, we have a few exceptions to that, e.g. when the App is just additional to CLI or if the GUI-application is non-native for OS X and/or hard to get in binary elsewhere (example: fontforge). Check out the homebrew-cask project if you’d like to brew native OS X Applications.
https://github.com/Homebrew/homebrew/blob/master/share/doc/homebrew/Acceptable-Formulae.md
Which could be a hold back.

@shinnok yep the test would normally replace the system "true" hack I put in.
wrt to the .app I think tarsnap gui would be OK. At worst, the same brew recipe can be hosted anywhere (like my gist) even in the tarsnap-gui project itself.

Per @cperciva's message on the mailing list, binary packages will be available in the future. At that point, Homebrew-Cask is probably the preferred way of making this installable via brew.

@dch There's now a -v and -h (cfc60c4fbf9b2629117a57616521f0d39f9516f8). Can you please refashion the brew recipe with these new options? :-)

updated, see above -- works for me -- thanks!!  NB before I can submit this, a new tag is needed with the -v option included. I'll keep an eye out for it ;-)
https://gist.github.com/dch/b39c65038f2b21314dbf

Good. That will be the v0.6 release. Let's wait until that is out.

On 16 Jun 2015, at 20:15, Dave Cottlehuber<EMAIL_ADDRESS>wrote:
updated, see above -- works for me -- thanks!! NB before I can submit this, a new tag is needed with the -v option included. I'll keep an eye out for it ;-)
https://gist.github.com/dch/b39c65038f2b21314dbf
—
Reply to this email directly or view it on GitHub.


[Bug]: list_dir does not work with recursive and filter_ext

📝 Description
If you use list_dir with recursive set to True and a set filter_ext attribute, then the function does not work.
It will first filter for the file extensions. However, if the first level consists of directories, then too much is filtered out at this point and it is no longer possible to search recursively for files.
🎯 Expected Behavior
It should be possible to filter by file extensions, even if the directory is to be searched recursively.
🐞 Current Behavior
As soon as a directory is searched recursively and filtered for a file extension, the function no longer works and returns an empty list.
🔁 Steps to Reproduce
Use list_dir with recursive set to True and filter for an file extension (for example '.png')
⚙ Technical Details
No response
💡 Suggestions or Possible Solutions
Filter for file extensions only at the end of the function
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Was fixed by #93

Was fixed by #93


feat(file-manager): add mkdir subtask

While discussing path handling, I noticed that our file manager lacks the ability to create a directory.
This feature would be useful when testing an application on crosscheck.
Supposedly, our file manager is able to do all the basic operations except this one.
I haven't figured out what to do with the points, so I took some from Create empty file (probably not the best option)

@AlreadyBored

Lgtm


Main loop empty batch behavior

Main loop should be configurable for cases of empty batches. I suggest the following:

Main loop checks the actual batch size
In the case of empty batch, acts accordingly to the configuration

ignore fetches another batch
warn prints a warning mesage but keep processing this (empty) batch
error prints an error message and terminates the whole training

we can handle error just by raising ValueError (it effectively stops the training and prints the error)
we should detect and raise an error on empty stream as well (i.e. no batches or empty batches only)

Btw, I am not sure if we agree on what is actually an empty batch. It is clear than an empty dict ({}) is indeed an empty batch. Can we check for anything else?

I think an empty batch is also a batch with zero elements in the sources.
E.g.
{
"ids": [],
"images": [],
"masks": [],
}

In these cases, I'd check one source for emptyness and assume that the others match.

@petrbel :+1: this is what cxtream empty batch looks like. I don't believe that {} should even be allowed, having the keys always there makes the code simpler.


«Convert» design of EventGroup page to the code

Rewrite this design into the code.

[x] Title
[x] Description
[x] Tags
[x] Calendar view
[x] Import button
[x] Import popup
[x] Favorite button

Convertation is completed! =)
If you have anything to say about this design, or you want to suggest changes, feel free to comment here.
https://github.com/one-zero-eight/InNoHassle-Website/assets/71718152/c371d346-e4d8-4a89-b85a-d76f88f84a0e


[resolved] incomplete file download ? file received capped at 1024MB then stopped.

the link to the questions is here requests response.iter_content() gets incomplete file ( 1024MB instead of 1.5GB )?
pasted from stackoverflow:
i have been using this code snippet to download files from a website, so far files smaller than 1GB are all good. but i noticed a 1.5GB file is incomplete
# s is requests session object
r = s.get(fileUrl, headers=headers, stream=True)

start_time = time.time()
with open(local_filename, 'wb') as f:
    count = 1
    block_size = 512
    try:
        total_size = int(r.headers.get('content-length'))
        print 'file total size :',total_size
    except TypeError:
        print 'using dummy length !!!'
        total_size = 10000000

    for chunk in r.iter_content(chunk_size=block_size):

        if chunk:  # filter out keep-alive new chunks

            duration = time.time() - start_time
            progress_size = int(count * block_size)
            if duration == 0:
                duration = 0.1
            speed = int(progress_size / (1024 * duration))
            percent = int(count * block_size * 100 / total_size)
            sys.stdout.write("\r...%d%%, %d MB, %d KB/s, %d seconds passed" %
                            (percent, progress_size / (1024 * 1024), speed, duration))

            f.write(chunk)
            f.flush()
            count += 1

using latest requests 2.2.1 python 2.6.6, centos 6.4
the file download always stops at 66.7% 1024MB, what am i missing ?
the output:
file total size :<PHONE_NUMBER>
...67%, 1024 MB, 5687 KB/s, 184 seconds passed

it seems the generator returned by iter_content() thinks all chunks are retrieved and there is no error. btw the exception part did not run, because the server did return the content-length in response header.
edit: after chaning my post url to something like
https://xxx.com/yyyfiles.php?userid=xxx&tutid=yyy&sessionid=zzz

the files downloads fine.

@oglops : I too facing same issue, Could you please add or update your code


Support PRs from forks

Pr built successfully 5 min after timeout (see tekton-pipelines status). Merging


Why does SlimPruner utilize the WeightTrainerBasedDataCollector instead of the WeightDataCollector before model compressing?

Because slim pruning needs to sparsify the BN weight (gamma) by adding a penalty item in original loss function, the sparsify process need a trainer to train the model.