OCI SELinux labeling mismatch when package only ships binary policy - greetd is broken

Hi OSTree team,
I am trying to debug an issue that surfaced in Ublue (https://github.com/ublue-os/main/issues/223). In Ublue we are making use of the OCI functionality heavily to produce custom downstream images of Fedora Silverblue. Some users have wanted to install greetd as their greeter for simplicity but whenever they do the resulting image is slightly broken because of wrong SELinux labeling.
I have investigated the package and I see that /usr/bin/greetd should be labeled as xdm_exec_t(see: here) which makes sense for a display manager but in the resulting images we see that /usr/bin/greetd is labeled as bin_t and therefore access gets denied by SELinux when the system is booted.
This is the output of ls -lZ in the images:
-rwxr-xr-x. 3 root root system_u:object_r:bin_t:s0 839488 Jan  1  1970 /usr/bin/greetd

this is the package source that shows the intended labeling:
I tried to trace down the issue and I found this function: sepolicy_from_base:
  1 fn sepolicy_from_base(repo: &ostree::Repo, base: &str) -> Result<tempfile::TempDir> {
  2     let cancellable = gio::Cancellable::NONE;
  3     let policypath = "usr/etc/selinux";
  4     let tempdir = tempfile::tempdir()?;
  5     let (root, _) = repo.read_commit(base, cancellable)?;
  6     let policyroot = root.resolve_relative_path(policypath);
  7     if policyroot.query_exists(cancellable) {
  8         let policydest = tempdir.path().join(policypath);
  9         std::fs::create_dir_all(policydest.parent().unwrap())?;
 10         let opts = ostree::RepoCheckoutAtOptions {
 11             mode: ostree::RepoCheckoutMode::User,
 12             subpath: Some(Path::new(policypath).to_owned()),
 13             ..Default::default()
 14         };
 15         repo.checkout_at(Some(&opts), ostree::AT_FDCWD, policydest, base, cancellable)?;
 16     }
 17     Ok(tempdir)
 18 }

I am not an expert but if I understand correctly this will look for the labels defined in /usr/etc/selinux in the resulting image and then it will restorecon based on that. In this case the package only ships a binary policy so my theory was that this would not find the rule /usr/bin/greetd			--	gen_context(system_u:object_r:xdm_exec_t,s0) which is defined in a binary policy therefore leading to the wrong labeling we observed.
We can use this minimal Containerfile to reproduce reliably:
FROM quay.io/fedora/fedora-silverblue:39
RUN rpm-ostree install -y greetd \
 && ls -lZ /usr/bin/greetd

Yes.  This relates to https://github.com/ostreedev/ostree-rs-ext/issues/388 where we want to basically move the labeling to the build time.
That said, something I think would also be interesting is to create a dedicated "image transform" container that can accept any arbitrary container as input and produce a "boot-ready" (including proper selinux labeling) container image.  This would replace a need for ostree container commit, and we could also do things like flatten layers in a configurable way and also (crucially) canonicalize timestamps etc.

To fix this in the current architecture (without changing ostree container commit) is messy because we'd have to:

fetch all the layers
compute the final merged filesystem tree
check out the final merged /etc/selinux at least
Use that policy to re-compute the final filesystem tree

Basically SELinux greatly complicates the architecture here because it's a set of files that can affect any other file.  So far the "always use the policy from the base" is a compromise.

I guess the first step towards the "container transformer" is to have a container that is capable of doing the labelling needed.
Mm I've been thinking about this and trying to get a container that allows me to properly label these files - even if it's just a PoC
I've been using podman and I ran into the issue that podman passes a context= option to the underlying overlayfs. This means inside the podman container no labelling actually works even if the container is privileged with label=nested.
I'm just leaving this comment here for my own reference but if anyone has any clues that'd be helpful.

An update here, we were able to troubleshoot around some non-filesystem related SELinux issues by shipping our custom policy and running semodule

Update: It works! You just have to run semodule -n -s targeted -X 200 -i /usr/share/selinux/packages/targeted/yourpolicyorwhereverelseitslocated.pp in the image in the CI as well. This is what the standard .rpm function %selinux_modules_install does as far as I can tell.

This specific policy does not have any changes to the filesystem so I am not sure if it would be affected by this but noting here just for reference.
In theory if can just load the policy from within the container this may work? (I didn’t know GitHub let you do that until today)
(sorry for the noise)

mm it seems that if we use podman btrfs storage driver, the overlayfs stuff gets entirely side-stepped and SELinux works in a rootless container even (we still need to pass /etc/selinux through as libselinux checks that)
podman run --rm -it --security-opt label=nested --privileged -v /etc/selinux:/etc/selinux greetd:bugx /usr/sbin/restorecon -RFv /usr/bin/greetd
Relabeled /usr/bin/greetd from system_u:object_r:container_file_t:s0:c1022,c1023 to system_u:object_r:xdm_exec_t:s0

so in theory if we can just grab a container, run it using the btrfs storage driver and just run restorecon -Rv / and save the resulting image then this should work?

so in theory if we can just grab a container, run it using the btrfs storage driver and just run restorecon -Rv / and save the resulting image then this should work?

Not really, ostree wants to own the security contexts, trying to change them behind its back won't really work.
The most reliable is going to be https://github.com/ostreedev/ostree-rs-ext/issues/510#issuecomment-1660662722 but it's ugly to implement today.

Yeah, this also breaks layering libvirt and enabling swtpm.  Annoying.

Does OSTREE just need the final /etc/selinux/targeted/context/files directory? Or do the files actually have to be applied as XAttrs within the image?

For swtpm, what would be the most minimal work around right now?
This breaks VMs quite a bit for me.

For my minimal workaround. I am using a systemd service to copy the swtpm binary to a mutable location. Bind mounting the mutable binary back over /usr/bin/swtpm. Then fix the file contexts and relabel it. What are the downsides for this work around?
# /etc/systemd/system/swtpm-workaround.service
[Unit]
Description=Workaround swtpm not having the correct label
ConditionFileIsExecutable=/usr/bin/swtpm
After=local-fs.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/swtpm-workaround

[Install]
WantedBy=multi-user.target

# /usr/local/bin/swtpm-workaround

#!/usr/bin/env bash

cp /usr/bin/swtpm /usr/local/bin/swtpm
mount --bind /usr/local/bin/swtpm /usr/bin/swtpm
semanage fcontext -a -t swtpm_exec_t "/usr/bin/swtpm"
restorecon /usr/bin/swtpm

ls -lZ /usr/bin/swtpm
-rwxr-xr-x. 1 root root system_u:object_r:swtpm_exec_t:s0 42136 Dec 14 10:21 /usr/bin/swtpm

This also relates to https://github.com/containers/bootc/pull/215 - that's where I'm thinking about going with this.

Another workaround is basically:
bootc usroverlay
restorecon /usr/bin/swtpm

@cgwalters, I am trying to get up to speed on this issue.  Do we have a settled way forward or is the solution TBD?

There are two potential fixes

"rechunker/rebuilder" which handles all this at buildtime and would fix other issues besides
Just hack this on the client side

Probably we need to do both. The latter is just ugly but doable.

My vote is the client side should be done first. Even if the rechuncker/rebuilder is amazing, it's not clear what percentage of users would use it consistently. $0.02

To xref, this test fixture demonstrates also customizing the selinux context for a toplevel: https://gitlab.com/fedora/bootc/tests/container-fixtures/-/merge_requests/13


Cache code size/hash in storage

Depends on https://github.com/rust-blockchain/evm/pull/140
Cache the code size/hash of an account in storage to avoid reading the full code to compute those values. Aims to reduce storage proof size.

Should I wait for an evm crate release or use git dependency? @sorpaas

@nanocryk can you merge master
@sorpaas can we get it reviewed and merged also ?

We shouldn't use lazy evaluation here. The security property is hard to guarantee. It's unsafe especially just around pre-/post-runtime upgrade time.
Please generate the code metadata on code creation instead. You can keep the current fallback code path just in case and/or use it in manual migrations.

Metadata is inserted in create_account which is "code creation". The fallback is necessary for code being deployed before this change.

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:
https://forum.polkadot.network/t/frontier-support-for-evm-weight-v2/2470/1

LGTM and Ci is green, @sorpaas can you merge it?

Sorry. Let me publish a new release (not yet, will ping you) and then change the evm version to that before merge. Otherwise things will be difficult to manage.

@sorpaas  before release a new version of evm, maybe we could release a new version of ethereum firstly?
https://github.com/rust-blockchain/ethereum/pull/50


Version 1.0

Adding initial version of the LiveRamp Identity Resolution and Translation native application quick start guide.

Shawn - this looks great! Because of the amount of code included here. Is it possible that you can provide a notebook of the code in different cells? Otherwise, this quickstart looks great!


⚠ Bofrost.fr has degraded performance

In 1b7875b, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 1888 ms

Resolved: Bofrost.fr performance has improved in 2c201db after 8 minutes.


Prepare master for next version

Prepare master for next version

@bot-gradle test and merge

Your PR is queued. See the queue page for details.

OK, I've already triggered a build for you.

Pre-tested commit build failed.

@bot-gradle test and merge

OK, I've already triggered a build for you.


Alternate Terrain Generation (ATG) support.

Somehow, the Custom Ore Generation settings aren't applied to ATG worlds in 1.7.10. I tested it with different settings, and nothing changed in the ore generation. I thought these two mods where compatible, but they don't seem to work together well... I am using version 1.2.26 with forge <IP_ADDRESS>8.

First time I've heard this is not working. Are you using the debugging wireframes? Are you sure there's no error in the log?

Nope. But with any other type than "vanilla" generation, it works (veins, layers, clouds, etc).
Somehow vanilla gen is broken. Tried it in vanilla worlds too, and it did the same.

Which "settings" are not having an effect?

The "vanilla" deposit setting.

Right, but what is "Vanilla" doing (or not doing)?  Are you not seeing any ore spawn?  Are you seeing too little?  Too much?  Or are you simply seeing the original ore spawning practices?

I looks like my size and frequency settings aren't being applied. Set them to X5.00 both, and the ore was spawning using regular X1.00 settings.

Ah, so the problem is that the main sliders aren't having an effect on the vanilla generation?


[Reflection] Check that the offset is within the section.

rdar://problem/49043621

@swift-ci test

@mikeash @adrian-prantl can you please take a look?

I'd probably have to look at the values in the debugger to be sure, but I don't think this is quite right. The FieldOffset value gets used to offset the RelativeDirectPointer that stores the field name. It's passed in as Offset here:
    if (FieldName)
      return (const char *)((uintptr_t)FieldName.get() + Offset);```

I think a better approach would be to have `getFieldName` take the expected bounds, and then it can check the result of its computation against those bounds before returning the string.

Build failed
Swift Test Linux Platform
Git Sha - 17685d34d1fea342be30047fbfa73b4ebf823528

@mikeash Implemented what you suggested, can you please take another look?

@swift-ci please test

Updated after conversation offline with Mike.

@swift-ci please test

@swift-ci please test

OK, this looks right now.

Thanks!


Write muldoon.met_timeseries.TemperatureTimeseries

I need a TemperatureTimeseries class for temperature time-series, similar to what I have for the PressureTimeseries class.
This class should include the ability to fit a Lorentz profile using a Gaussian processes approach since the temperature data (at least for Mars 2020 MEDA ATS) have a lot of wiggle in them.

Should also probably include an associated collection of the best-fit vortex model parameters - PressureTimeseries.popts and PressureTimeseries.uncs.


Recall() feature — transferring Orb back to contract after Oath expires

text is reworded Eric's message
The way Orb works currently is that you swear an Oath for a certain period, and after this period is over (or even before) you can extend the honoredUntil param and after it is over you can reswear the Oath with some new params too (like different Harberger tax etc).
However, with "trial mode" (short Oath period), we realized this is not a great design. Creator will now have to auction off his Orb at trial value, and cannot get any of the auction sale value out of swearing a 10 year Oath. So we fear that this will start leading to situations where creator maybe sets the Harberger tax rate very high (because why not) and then he buys it back, and then he lowers the tax rate and does a real auction.
The danger with this is that the person who bought the trial Orb can feel rugged.
The new idea is to allow creator to recall the Orb after the initial trial period. This will be communicated clearly to the person who buys it in trial mode — after the trial period is up, they will lose the Orb (almost guaranteed). We did not think about this option initially, because we didn't consider that the Orb will still get traction while in trial mode. Actually some users will prefer to buy the cheap trial Orb if they have just a few questions to the creator, than having to win against long-term Orb speculators.
In order to introduce the recall feature, a code change is necessary.

This commit adds recall() function that allows the Orb creator to transfer the Orb back to the contract after Orb Oath expires. It keeps remaining Keeper funds, allowing for withdrawal later.

This looks good to me, nice thinking for not force withdrawing funds for keeper.


Added F# iOS Sample

Continuation of https://github.com/dotnet/net6-mobile-samples/issues/88
Decided to try it out and it worked out pretty easily.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: delneg sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

I can confirm it didn't break anything,
 dotnet build fsharp/HelloiOSFSharp -t:Run ran fine.
Thanks!


Add 'Sound'  to rfxtrx DATA_TYPES

fixes reception of Selectplus and auto adding chime sensor.

I think you have made a mistake with git merge. You PR don't change only you file and can not be merget with dev. Make a fresh fork and a new cleanup PR...

sorry. made a new pull-request
https://github.com/home-assistant/home-assistant/pull/2477


Add forge cleanup

It's been inspired by and named after a projectile-cleanup-known-projects. But unlike it's predecessor it is executing in a separate process to not to block UI. Because of that I had to made a few trade-offs that I tried to describe in comments in code.
The feature has been motivated by my recent realisation that I have done PRs well over 150 different repositories for last year. That made searching for a project a bit cumbersome, even with a help of helm-projectile. Since I was not expecting to do any further PRs to around a 100 of these repositories so I deleted them, then run projectile-cleanup-known-projects. However, there's no counterpart for forge database for that. The list from forge-list-repositories was long and I didn't had a method to clean it nicely like I've done for the projectile.
Note that this branch includes the on from #184.

TBH I learned that when writing this too. Still not fully grasping how's pcase-let is doing the destructurizing bindings, but will get there eventually.

TBH I learned that when writing this too. Still not fully grasping how's pcase-let is doing the destructurizing bindings, but will get there eventually.


Create protobuf message and service for model shards

Create ModelShard protobuf message.
Create ModelShards service to create, stream, and delete model shards.
Update BUILD and all_protos files to include new ModelShards service.

This change is


7-6a

Compared to CLRS2e's 7-5, works on a subarray A[p:r] and deines a sequence of sorted elements of that subarray, instead of defining the sorted output array A'.


Properly copy attributes before modifying in JSQMessagesCollectionViewFlowLayout

Pull request checklist

[x] All tests pass. Demo project builds and runs.
[x] I have resolved any merge conflicts.
[x] I have followed the coding style, and reviewed the contributing guidelines. Confirmation: ____

This fixes issue #1218
What's in this pull request?
Ensure that attributes array is properly copied before modification.
Could also fix #1424 but I'm not sure about that.

Current coverage is 61.27% (diff: 50.00%)

Merging #1868 into develop will not change coverage

@@            develop      #1868   diff @@
==========================================
  Files            62         62
  Lines          2249       2249
  Methods         608        608
  Messages          0          0
  Branches        135        135
==========================================
  Hits           1378       1378
  Misses          797        797
  Partials         74         74

Powered by Codecov. Last update 624fcfa...51a0b65

Thanks @algrid !
Based on the discussion in #1218 -- I'm not completely sure how a deep copy resolves this. Can you explain more? Do you have before/after repro steps?

@jessesquires I can reproduce the original problem in the demo app:

Turn on 'Springy bubbles' in Setting
Push via storyboard
Start scrolling

I get the following in my console:
2016-10-17 22:46:49.101 JSQMessages[5042:411339] Logging only once for UICollectionViewFlowLayout cache mismatched frame
2016-10-17 22:46:49.102 JSQMessages[5042:411339] UICollectionViewFlowLayout has cached frame mismatch for index path <NSIndexPath: 0xc000000004200016> {length = 2, path = 0 - 33} - cached value: {{4, 3421}, {367, 143}}; expected value: {{4, 3423}, {367, 143}}
2016-10-17 22:46:49.102 JSQMessages[5042:411339] This is likely occurring because the flow layout subclass JSQMessagesCollectionViewFlowLayout is modifying attributes returned by UICollectionViewFlowLayout without copying them

With this PR the problem is gone.
I'm doing here exactly what the log message suggests - don't modify attributes returned by UICollectionViewFlowLayout, instead make copies. Inspired by this: http://stackoverflow.com/a/33172179

Thanks @algrid ! 💯


[Feature request] bit offsets for binary templates

There's no way to deal with anything that's not byte aligned with "binary" templates, unless I'm missing something. It would be useful for reading files that contain bitstreams, and also raw devices.

Could you provide an example of how you'd use it, maybe a specific data format?
The expr Tcl command has bitwise operator support, so it could be done with that and Hex Fiend's uint8 command.
This was also mentioned here:
https://github.com/ridiculousfish/HexFiend/issues/89#issuecomment-399803545

Since HF's templates were inspired by WinHex, in their manual they describe a "uint_flex" command:

A special variable type supported by templates is uint_flex. This type allows to compose an unsigned integer value from various individual bits within a 32-bit (4-byte) range in an arbitrary order and is even more flexible than a so-called bit field in the C programming language.
uint_flex requires an additional parameter string in inverted commas that specifies exactly which bits are used in which order, separated by commas. The bit listed first becomes the most significant bit (high value bit) in the resulting integer, and it is not interpreted as a + or - indicator. The bit listed last becomes the least significant bit in the resulting integer.
The bits are counted starting with 0. Bit 0 is the bit that is the least significant bit of the 1st byte. Bit 31 is the most significant bit of the fourth byte. Thus, the definition is based on little-endian philosophy.
For example,
uint_flex "15,14,13,12,11,10,9,8,7,6,5,4,3,2,1,0" "Standard 16-bit integer"
is exactly the same as uint16, the common unsigned 16-bit integer variable.
uint_flex "31,30,29,28,27,26,25,24,23,22,21,20,19,18,17,16,15,14,13,12, 11,10,9,8,7,6,5,4,3,2,1,0" "Standard 32-bit integer"
is exactly the same as uint32, the common unsigned 32-bit integer variable.
The benefit of uint_flex, though, is that the number, the position, and the usage order of all bits can be chosen arbitrarily. For example,
uint_flex "7,15,23,31" "An unusual 4-bit integer"
composes a 4-bit integer out of the respective most significant bits of each of the four bytes involved. If these four bytes happen to be
F0 A0 0F 0A =
11110000 10100000 00001111 00001010,
bit 7 is 1, bit 15 is 1, bit 23 is 0, and bit 31 is 0.
So the resulting uint_flex is 1100 = 18 + 14 + 02 + 01 = 12.

maybe something like this would work?

I think instead of a single type that reads only 4 bytes, this could be implemented with 8, 16, 32, and 64 bit size commands. I also think "bits" might be better name suffix. For example uint8_bits.

Could you provide an example of how you'd use it, maybe a specific data format?

Sure, if you were to parse the encoded bitstream of media files, there's not so much a file structure, but a stream format, which usually needs to be decoded an arbitrary number of bits at a time.
Here's the start of a core frame header for DTS as an example.
....7FFE 8001FC3C
7DB27702 0D3A0009
EF7BDEDE F7BD8000

7FFE8001 is the sync word. Then there are
FTYPE(1)
SHORT(5)
CPF(1)
NBLKS(7)
FSIZE(14)
...

The sizes are in bits.
Yes,  I could probably read a uint8 then and mask and shift to get the component bits.
(0xFC & 0x80)  >> 7 would get me 1 for FTYPE
(0xFC & 0x7C) >> 2 for 31 as SHORT
(0xFC & 0x02) >> 1 is CPF...
and it's almost feasible till here where I need to read 7 bits of NBLKS across two bytes. It is still doable, I could "carry over" the bit, go back and read two bytes this time but at this point it's like I'm trying to use a tape measure when what I really need is a set of feeler gauges.
More realistically, I would keep a secondary buffer and offset on my own, and I realize any implementation of a feature like this would have to involve a similar mechanism, but I found myself wishing this was a thing often enough and wondered if I was alone.
As far as how one would use such a feature, I have to admit I'm not entirely sure what the best way would be, it shouldn't be the "default" for sure, maybe a command to read n bytes and initialize a sort of sub-context to read them into a struct with bit fields? something like

unpack_bytes size {
    bit count label
    ...
}

I added an experimental uintX_bits command based on uint_flex described above. I also released a new beta for testing: https://github.com/ridiculousfish/HexFiend/releases
The command works the same as above but has 4 variants: uint8_bits, uint16_bits, uint32_bits, and uint64_bits for reading respective number of bytes.
The second argument is a string with the bit offsets separated by commas. The command only returns a single bitwise or'd value from the bits, starting from left to right. The integer is read via the current endian, so to read from right to left in bits you must read as big endian. The last argument is the label, same as other type commands.
So for your example, the fields could be read like so:
big_endian
uint32_bits "31" "FTYPE"
move -4
uint32_bits "30,29,28,27,26" "SHORT"
move -4
uint32_bits "25" "CPF"
move -4
uint32_bits "24,23,22,21,20,19,18" "NBLKS"
move -4
uint32_bits "17,16,15,14,13,12,11,10,9,8,7,6,5,4" "FSIZE"

However, looking at the larger example of DTS here: http://stnsoft.com/DVD/dtshdr.html I can see how this would be kind of clunky to work with, especially since in that case it's using 6 bytes and nothing is on byte boundaries. Your example could be another way to make it work. For example if a "bits" command was added it could keep track of the current bit offset and allow reading n number of bits for more easily handling of a stream of bits back to back:
bits 1 "ftype"
bits 5 "short"
bits 1 "cpf"
bits 7 "nblks"
bits 14 "fsize"
bits 6 "amode"
bits 4 "sfreq"
bits 5 "rate"
bits 1 "mix"
bits 1 "dynf"
bits 1 "timef"
bits 1 "auxf"
bits 1 "hdcd"

It could keep track of the bit offset, with any other read command resetting the bit offset back to 0.
I think both formats would be useful for different scenarios.

Kaitai Struct uses the second approach: http://doc.kaitai.io/user_guide.html#_bit_sized_integers
The parsing is fixed from left to right, which makes sense for bitstreams. In my opinion, your second approach looks nicer and the template is easier to write and to maintain (especially, if you forget a bit).

I came across a use case where I need to be able to read some data bits. Is this something that is still being worked on?

Thanks, the flex bits command saves a lot of tedious math needing to be copy pasted in every template. I had a hard time figuring out the bit ordering for the multiple byte variants, though, uint16_bits numbering 7~0 then 15~8 was impossible to figure out without reading the source. After seeing how it's implemented, it seems somewhat natural, but I think adding the logic to treat those bytes as a continuous sequence of bits would be worthwhile.

I think as a feature this can be considered implemented, maybe could see an enhancement in the future, but once you get used to the bit and byte ordering, it is a pretty solid answer to what was missing.


cop: set the error field properly for the batch cop task response

Signed-off-by: cfzjywxk<EMAIL_ADDRESS>
What is changed and how it works?

Issue Number: Close #13856
What's Changed:

Fill the batch task response error field properly, the error field should be None if no error happens.

Related changes
Check List
Tests

Unit test

Side effects
Release note
None

/merge

/test


Using plugin without registering account

Is there a way to use plugin without registering an account?

Thank you for raising this issue! Initially, the plugin was designed to work without requiring user registration. However, during the official Obsidian plugin review process, we were asked to include a registration flow for better user management and plugin compliance (as detailed in [this PR](https://github.com/obsidianmd/obsidian-releases/pull/4667)).
Additionally, user registration helps us manage and protect ownership of custom domains, ensuring early adopters’ rights and preventing conflicts in domain allocation.
We understand the desire for a seamless, account-free experience, and we’ll consider adding an optional "guest mode" in future releases to address this need. Stay tuned, and we welcome any further feedback! 😊


Does this library work with pre-recorded videos?

I read through the documentation and based on my understanding the library works with live video feeds. I am interested to know if it can be passed a video file as input as well?
Thanks

Yes! It uses OpenCv for video capturing which can be used for both pre-recorded videos or live videos. Just change the 0 in cv2.VideoCapture(0) to the filename in quotes and it will work.

Thank you!
I tested it as you suggested (eye_tracker.py in particular). The video is playing back slower than the original - this is not an issue but was curious to see if it's just me.
I don't see any of the outputs that I normally see with the livestream (e.g Looking up, down) in the console when using a pre-recorded video as the input. Can you please let me know if it's a bug or I am doing something wrong?


[Quickdraw] I am here to learn how to open and close issues on Github! 🤠

I am here to learn how to open and close issues on Github! 🤠
I will close it in 10 seconds 🤞

I fixed my issue


building with crystal with --debug errors out

vagrant@vagrant-ubuntu-trusty-64:~/crystal$ ./bin/crystal build --debug src/compiler/crystal.cr
Using compiled compiler at .build/crystal
.crystal//home/vagrant/crystal/src/compiler/crystal.cr/main.o: file not recognized: File truncated
collect2: error: ld returned 1 exit status
Error: execution of command failed with code: 1: `cc -o crystal .crystal//home/vagrant/crystal/src/compiler/crystal.cr/main.o .crystal//home/vagrant/crystal/src/compiler/crystal.cr/
...

Duplicate of #257
For now we have to use the old puts debugging style.


added user-agent header to search request to fix 403 issue

The how long to beat search call seems to result in a 403 error if the user-agent request header is 'axios' which axios appears to set by default. I added the user agent header in the same way it is being done on the detailHtml call and it seems to work.

I changed src/main/hltbsearch.ts 80th line from node modules in my project added new user agents (Mozilla/5.0) but still getting 403. Am i missing something else?

Yep, sorry I forgot to push a few other changes. If you are modifying in your local node modules you'll need to add the user agent header to dist/main/hltbsearch.js

Yep. I missed that one too. Now works as expected, Thanks.

Thank you @bigjca, I just added the line (since I obviously only forgot to set the user-agent properly when searching).
I will close this PR, since you must not modify the js files in dist folder (they are generated by tsc). I am still lacking to convert this to github actions, making the build process automated (as well as the publish process).
v1.7.1 is now available, please test and enjoy
https://www.npmjs.com/package/howlongtobeat


🐞 : Error: Expected a value of type 'Map<String, dynamic>', but got one of type 'String' Error

Full error while runnig my flutter authkit app
Launching lib/main.dart on Chrome in debug mode...
package:teletest/main.dart:1
This app is linked to the debug service: ws://<IP_ADDRESS>:41535/g1YH8UNqwtA=/ws
Debug service listening on ws://<IP_ADDRESS>:41535/g1YH8UNqwtA=/ws
💪 Running with sound null safety 💪
Connecting to VM Service at ws://<IP_ADDRESS>:41535/g1YH8UNqwtA=/ws
Flutter Web Bootstrap: Programmatic
Failed to load font Roboto at https://fonts.gstatic.com/s/roboto/v20/KFOmCnqEu92Fr1Me5WZLCzYlKw.ttf
TypeError: Failed to fetch
Error: Expected a value of type 'Map<String, dynamic>', but got one of type 'String'
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 266:49      throw_
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 99:3        castError
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/operations.dart 452:10  cast
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/classes.dart 635:14     as_C
packages/appwrite/services/account.dart 23:27                                     get
dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 45:50                <fn>
dart-sdk/lib/async/zone.dart 1653:54                                              runUnary
dart-sdk/lib/async/future_impl.dart 147:18                                        handleValue
dart-sdk/lib/async/future_impl.dart 766:44                                        handleValueCallback
dart-sdk/lib/async/future_impl.dart 795:13                                        _propagateToListeners
dart-sdk/lib/async/future_impl.dart 566:5                                         [_completeWithValue]
dart-sdk/lib/async/future_impl.dart 639:7                                         callback
dart-sdk/lib/async/schedule_microtask.dart 40:11                                  _microtaskLoop
dart-sdk/lib/async/schedule_microtask.dart 49:5                                   _startMicrotaskLoop
dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 166:15               <fn>

my pubspec.yaml :
name: teletest
description: A new Flutter project.
version: 1.0.0+1

environment:
  sdk: '>=2.18.5 <3.0.0'

dependencies:
  flutter:
    sdk: flutter
  appwrite_auth_kit: 0.7.0
  random_string: ^2.3.1

dev_dependencies:
  flutter_test:
    sdk: flutter

flutter:
  uses-material-design: true

Also getting this error sometimes while running the app after cleaning up the project with flutter clean and then running it.
Launching lib/main.dart on Chrome in debug mode...
lib/main.dart:1
This app is linked to the debug service: ws://<IP_ADDRESS>:36417/7MuvaS9DfcQ=/ws
Debug service listening on ws://<IP_ADDRESS>:36417/7MuvaS9DfcQ=/ws
💪 Running with sound null safety 💪
Connecting to VM Service at ws://<IP_ADDRESS>:36417/7MuvaS9DfcQ=/ws
Flutter Web Bootstrap: Programmatic
wasm streaming compile failed: TypeError: Could not download wasm module
falling back to ArrayBuffer instantiation
Aborted(both async and sync fetching of the wasm failed)
failed to asynchronously prepare wasm: RuntimeError: Aborted(both async and sync fetching of the wasm failed). Build with -sASSERTIONS for more info.
Aborted(RuntimeError: Aborted(both async and sync fetching of the wasm failed). Build with -sASSERTIONS for more info.)
RuntimeError: Aborted(both async and sync fetching of the wasm failed). Build with -sASSERTIONS for more info.
<EMAIL_ADDRESS>159:75   Pa
<EMAIL_ADDRESS>159:480  Bb
<EMAIL_ADDRESS>160:259  <fn>

@hemangjoshi37a I don't think the error is related to this package.


Update logging to use @google-cloud/logging-winston

cc @jmdobry

Travis failures are due to this issue.


Sign out method errors out

I integrated the WorkOS into a Next.js application using the tutorial on the docs site. The Sign In and the Sign Up links work and I get redirected automatically to my app where the user is shown as logged in. I am able to fetch the user details and show them. However, the signOut method errors out with the following page:

If I manually navigate back to the application, the user is correctly shown as signed out, despite the error above.
My sign out logic is simple. I'm only calling the WorkOS signOut method in a server action and not doing anything else.
        <form
          action={async () => {
            "use server";
            await signOut();
          }}
        >
          <DropdownMenuItem asChild>
            <button className="w-full cursor-pointer" type="submit">Log out</button>
          </DropdownMenuItem>
        </form>

Is this a bug? Or do I need to complete some kind of setup in my WorkOS dashboard?

This is happening because you haven't configured a homepage URL in your WorkOS dashboard. Once you set up a homepage URL under the "Redirects" section in the dashboard it will work as intended.


Delete process data

Adding Processed data deletion, both to the qiita_db and the GUI.

Coverage decreased (-0.06%) to 79.12% when pulling 2fa574ecec632a0472a662f9c6daa7649b4c2d43 on antgonza:delete-process-data into 0744eed0048002fbdb8c1ee9d35bc9101d08ab67 on biocore:master.

Coverage decreased (-0.08%) to 79.1% when pulling d63fb3eaf7daa113cac1105bb3a0771ccd6a5774 on antgonza:delete-process-data into 0744eed0048002fbdb8c1ee9d35bc9101d08ab67 on biocore:master.

Few comments. Will test interface once addressed.

Coverage decreased (-0.08%) to 79.1% when pulling 99f196f9faaf0dcfd59ae531d351a58cf9027721 on antgonza:delete-process-data into 0744eed0048002fbdb8c1ee9d35bc9101d08ab67 on biocore:master.

Ready for another review.

Few more comments @antgonza. Thanks for putting this together!

:+1:

Coverage decreased (-0.06%) to 79.12% when pulling c195dade3cf62ba527c01de28fd36a94d795c217 on antgonza:delete-process-data into 0744eed0048002fbdb8c1ee9d35bc9101d08ab67 on biocore:master.


Updated PHONY to have the command on a separate line

@dgn please try this and let me know if it works. Both work on my system.

Yeah, that's what I'm thinking too. Does it build for you @jwendell ?

I didn't try, but I don;t think your wrong syntax would cause a failure. It would not just work as intended.

Thank you.


Dropbox integration stopped working

Looks like an update of django-storages has broken Dropbox integration.
Package versions:
django-dbbackup==3.1.3
django-storages==1.6.3
dropbox==8.0.0

Stack trace:
(foobar-env) foobar@main:~/baz-website/foobar$ python3 manage.py mediabackup --compress
Writing file to main.foobar.org-2017-07-08-175710.tar.gz
TypeError: expected request_binary as binary type, got <class 'django.core.files.base.File'>
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dbbackup/utils.py", line 111, in wrapper
    func(*args, **kwargs)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dbbackup/management/commands/mediabackup.py", line 52, in handle
    self.backup_mediafiles()
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dbbackup/management/commands/mediabackup.py", line 103, in backup_mediafiles
    self.write_to_storage(tarball, filename)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dbbackup/management/commands/_base.py", line 88, in write_to_storage
    self.storage.write_file(file, path)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dbbackup/storage.py", line 82, in write_file
    self.storage.save(name=filename, content=filehandle)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/storages/backends/dropbox.py", line 111, in _save
    self.client.files_upload(content, self._full_path(name))
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dropbox/base.py", line 1267, in files_upload
    f,
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dropbox/dropbox.py", line 234, in request
    timeout=timeout)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dropbox/dropbox.py", line 325, in request_json_string_with_retry
    timeout=timeout)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dropbox/dropbox.py", line 369, in request_json_string
    type(request_binary))

Traceback (most recent call last):
  File "manage.py", line 24, in <module>
    execute_from_command_line(sys.argv)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dbbackup/utils.py", line 111, in wrapper
    func(*args, **kwargs)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dbbackup/management/commands/mediabackup.py", line 52, in handle
    self.backup_mediafiles()
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dbbackup/management/commands/mediabackup.py", line 103, in backup_mediafiles
    self.write_to_storage(tarball, filename)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dbbackup/management/commands/_base.py", line 88, in write_to_storage
    self.storage.write_file(file, path)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dbbackup/storage.py", line 82, in write_file
    self.storage.save(name=filename, content=filehandle)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/storages/backends/dropbox.py", line 111, in _save
    self.client.files_upload(content, self._full_path(name))
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dropbox/base.py", line 1267, in files_upload
    f,
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dropbox/dropbox.py", line 234, in request
    timeout=timeout)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dropbox/dropbox.py", line 325, in request_json_string_with_retry
    timeout=timeout)
  File "/home/foobar/.env/foobar-env/lib/python3.5/site-packages/dropbox/dropbox.py", line 369, in request_json_string
    type(request_binary))
TypeError: expected request_binary as binary type, got <class 'django.core.files.base.File'>

See also:

https://github.com/jschneier/django-storages/issues/227
https://github.com/jschneier/django-storages/issues/301

Hello @maxmalysh
Is it really related to Dbbackup ?

I was having the same problem, it worked again when I updated these libs to a certain version.
dropbox==7.3.0
django-storages==1.5.2

I am still having this issue with:
dropbox==8.0.0
django-storages==1.6.5

@JakobGM  Is it related to DBBackup ?

@ZuluPro I am not sure exactly where the issue lies, as I am unable to isolate the issue. But the exception is raised within the dropbox module, so it might not be related directly to DBBackup.

Here is my exception:
TypeError: expected request_binary as binary type, got <class 'django.core.files.base.File'>
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "dbbackup/utils.py", line 111, in wrapper
    func(*args, **kwargs)
  File "dbbackup/management/commands/dbbackup.py", line 61, in handle
    self._save_new_backup(database)
  File "dbbackup/management/commands/dbbackup.py", line 88, in _save_new_backup
    self.write_to_storage(outputfile, filename)
  File "dbbackup/management/commands/_base.py", line 88, in write_to_storage
    self.storage.write_file(file, path)
  File "dbbackup/storage.py", line 82, in write_file
    self.storage.save(name=filename, content=filehandle)
  File "django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "storages/backends/dropbox.py", line 111, in _save
    self.client.files_upload(content, self._full_path(name))
  File "dropbox/base.py", line 1267, in files_upload
    f,
  File "dropbox/dropbox.py", line 234, in request
    timeout=timeout)
  File "dropbox/dropbox.py", line 325, in request_json_string_with_retry
    timeout=timeout)
  File "dropbox/dropbox.py", line 369, in request_json_string
    type(request_binary))

@ZuluPro I've found the error, it's inside django-storages. I'll provide a fix soon.

Hi,
Any news regarding this issue?

pip uninstall django-storages
pip install -e 'git+https://github.com/jschneier/django-storages.git#egg=django-storages'


Integrate with wasmtime; 集成 wasmtime 运行时

1. Background
Currently, Layotto supports load and run *.wasm files by integrating wasmer-go and wasmer, now we want to support wasmtime as another WebAssembly Runtime.
2. Why is this needed?
Compared with wasmer, wasmtime has more advanced features, which allow us to make more settings for the running *.wasm files. For details, please refer to #607
3. draft design

Related code paths

Reference Code

MOSN has provided extension points. the interface is located in the file: pkg/types/wasm.go. In theory, we can use the wasmtime API to implement several interfaces in Layotto.
4. Other references

https://github.com/bytecodealliance/wasmtime
https://github.com/bytecodealliance/wasmtime-go

中文
一、背景
目前 Layotto 是通过集成 wasmer-go 跟 wasmer 来支持 *.wasm 文件的加载运行，现在我们希望支持 wasmtime 来加载运行 *.wasm 文件
二、意义&价值
相比于 wasmer，wasmtime 具备更多的高级特性，可以让我们对运行着的 *.wasm 文件做更多设置，具体可以参考 #607
三、设计草案

相关代码路径

参考代码

MOSN 已经留好了扩展点，接口所在文件：pkg/types/wasm.go，理论上应该在 Layotto 中结合 wasmtime 的 API 实现里面的几个接口即可。
四、其他参考资料

https://github.com/bytecodealliance/wasmtime
https://github.com/bytecodealliance/wasmtime-go

/assign

进展：

不需要写到 mosn 仓库里，可以注册进去
函数有差异，有适配成本

进展:
因为wasmtime对function注册相关的支持没有wasmer完备，还在单元测试

目前集成wasmtime单测通过，集成测试有点问题，给bytecodealliance提了issue
https://github.com/bytecodealliance/wasmtime-go/issues/145

https://github.com/bytecodealliance/wasmtime-go/issues/146

目前demo中function2可以正常加载及运行，function1有点问题
https://github.com/bytecodealliance/wasmtime-go/issues/149

https://github.com/bytecodealliance/wasmtime/issues/4194


Expose the graphql operation as a constant in the generated genqlient file

This allows client code to see the operation (query or mutation) exactly as genqlient sends it over the wire.  This data was already available in the generated safelist.json file, but now it's easily available from Go code as well.
Fixes #236
I have:

[x] Written a clear PR title and description (above)
[x] Signed the Khan Academy CLA
[x] Added tests covering my changes, if applicable
[x] Included a link to the issue fixed, if applicable
[x] Included documentation, for new features
[x] Added an entry to the changelog

This looks good to me!
Note that the constant name could conflict with a separate operation with the same name, e.g. getViewer and getViewerOperation (which would have the operation string getViewerOperationOperation). Such a case seems very unlikely, though (and is definitely confusing), so I don't think we need to worry about it.

Yeah, the same is currently true of getViewerResponse as well.  I agree we probably don't need to worry about it.


Templates which have note overview plugin code in them do not work

I tried creating a template which has a note overview (from note overview plugin) to get summary of tasks I have not completed. If you look at the code below the noteview in the template has {{notebook}} {{title}}, {{tags}} etc these get removed when I create note from template
This is the template:
---
project: dropdown(Work, Personal)
template_title: {{ subproject }}
name: text
template_tags: {{#custom_datetime}}YYYY{{/custom_datetime}}, {{#custom_datetime}}MMMM{{/custom_datetime}},{{ project }}

---

## {{ template_title }} - {{  name  }}

This note contains the meeting minutes of the weekly meet held on {{ datetime }} for {{ project }}.

This is how it transforms when I create a note from template
 Test

This note contains the meeting minutes of the weekly meet held on 24/09/2021 10:47 for Work.

Yup, this is expected because the text of form {{ something }} is considered as variable by this plugin. But I'll explore if there's a way to allow users to use these braces as normal text.

Hello,
I ran into the same issue and found two possible workarounds. Here they are in case someone can find this useful.
Solution 1: Defining text with template custom variable

Define a template custom variable as a dropdown variable

e.g.: note_text: dropdown({{title}} ({{created_time}}))
By using a dropdown variable with a single choice it is filled automatically when creating a new note from template

In the note-overview section, use the variable defined above

listview:
  text: "{{note_text}}"

Solution 2: Escaping double curly brackets and defining note-overview-plugin as template custom variable

Similar principle as the first solution, but applied differently
Define a dropdown variable note_overview: dropdown(note-overview-plugin)
Define the note-overview section with


API View does not show methods/models correctly due to namespace check

API view does not show aio methods and models for https://pypi.org/project/azure-search-documents/11.1.0b4/#files

@xiangyan99 please confirm if this is still the case

This seems to be addressed:

@xiangyan99 do you agree?

I think this has been addressed. Please re-open if you feel it is still an issue.


Download stalls of Last Few Images

I have noticed that sometimes the app will freeze and not download anymore of several final items, ex: Found 474 Downloaded 468. Please implement some method of retrying images rather than stalling these downloads.

Thank you for your feedback @apoapostolov! I'm currently working on timeout-retrying method.


build istio gateway by using dynamic go client

I used unstructured.Unstructured for every thing until my istio gateway.
I used "kubectl api-resources | grep gateway" to obtains information relatives to the schema,
kubectl api-resources | grep gateway
gateways                          gw           networking.istio.io/v1beta1            true         Gateway
but at the end, I have this error
Failed to create gateway: the server could not find the requested resource
Can someone help me?
Here is my function,
func (c *K8sDynamicClient) CreateGateway(name, namespace string, ingress string, host string) {
gatewayRes := schema.GroupVersionResource{
Group:    "networking.istio.io",
Version:  "v1beta1",
Resource: "gateways",
}
    gatewayObject := &unstructured.Unstructured{
            Object: map[string]interface{}{
                    "apiVersion": "networking.istio.io/v1beta1",
                    "kind":       "Gateway",
                    "metadata": map[string]interface{}{
                            "name":      name,
                            "namespace": namespace,
                    },
                    "spec": map[string]interface{}{
                            "selector": map[string]interface{}{
                                    "istio": ingress,
                            },
                            "server": []map[string]interface{}{
                                    {
                                            "hosts": []string{
                                                    host,
                                            },
                                            "ports": map[string]interface{}{
                                                    "name":     "http",
                                                    "number":   80,
                                                    "protocol": "HTTP",
                                            },
                                    },
                            },
                    },
            },
    }

    gateway, err := c.Client.Resource(gatewayRes).Create(context.TODO(), gatewayObject, metav1.CreateOptions{})
    if err != nil {
            log.Fatalf("Failed to create gateway: %v\n", err)
    }

}

Gateways are a namespaced resource, your client needs to set Namespace(namespace) to make it submit the request to a namespaced endpoint

Thx a lot Jordan, you unlocked my problem.


Problem: Gravity bridge orchestrator always run with relayer

Solution: Add relayer flag to orchestrator and gorc

https://github.com/PeggyJV/gravity-bridge/pull/210
It seems implemented in upstream.


Update GEP-713 and add BackendTLSPolicy implementation

What type of PR is this?
/kind feature
/kind gep
What this PR does / why we need it:
This updates GEP-713 with the information needed for BackendTLSPolicy's status (including making this a new recommendation for Policy status), and also implements BackendTLSPolicy.
Which issue(s) this PR fixes:
Updates #713
Updates #1897
Does this PR introduce a user-facing change?:
BackendTLSPolicy has been added to control TLS from Gateway to backends specified in HTTPRoutes.

This needs a little more work to add tests for the BackendTLSPolicy CEL rules, but I haven't done that before, not sure how long it will take. This will let folks review the rest while I work on that.
/hold

BackendTLSPolicy also has the problem that support for it should be Extended, but it will be tricky to write conformance tests for. I probably also need to add something to GEP-1897 about how implementations should signal that they support the new object.
I'll work on that now.

I voiced concerns about this one at the v1.0 API review. Overall, those concerns kind of boil down to the backend TLS and ancestors feeling even more experimental than GEP-713 itself, which worries me: landing this will fold those into GEP-713, and no one reading the GEP later will know the gradations between bits of 713.
But perhaps I should trust the experimental process more. @youngnick, I'm OK landing this as experimental for v1.0. Probably sometime after that we should take a serious look at graduation criteria for GEP-713 -- it's significant enough that we might not want to leave it exempt.

/lgtm

/hold cancel


Add IndexedVectorizedOapRecordReader

What changes were proposed in this pull request?
#536 task 5

ColumnarBatch add 3 method:

boolean isFiltered(int rowId)  to  determine row is filtered by index

void markAllFiltered() to mark all rows filtered

void markValid(int rowId) to mark row is valid

Add IndexedVectorizedOapRecordReader let Indexed read can use Vectorized Read

Add test case to ColumnarBatchSuite to test ColumnarBatch 3 new method

Add test case named IndexedVectorizedOapRecordReader - direct path read to test IndexedVectorizedOapRecordReader

5.enable batch.isFiltered(i) assertion in VectorizedOapEncodingSuite
How was this patch tested?
ColumnarBatchSuite & VectorizedOapIOSuite.scala

@chenghao-intel this pr is task 5 of #536， and #547 is all code,  @linhong-intel @Moonlit-Sailor @xuanyuanking already reveiw #547, so I split #547 to small prs to sumbit~

@chenghao-intel The next pr will be glue code to let OAPFileFormat to use full scan VectorizedOapRecordReader or Indexed VectorizedOapRecordReader for Parquet read.

LGTM

There are a few style issues. Can you please take care all of them?

@carsonwang I am a little confusion. Java needs 4 spaces in the method declaration, and and two spaces on the method body . should we enhance checkstyle-plugin to help us to control the code format, particularly Java Code~

@carsonwang Are there any other style problems?

The style looks good to me now. Thanks @LuciferYang

Thanks @carsonwang , already fix ‘two spaces style',  there are some java code already merged have code style problems, I create #599  to fix this problems, but I strongly suggest that we can provide the formatting template available for Intellij or use maven-checkstyle-plugin to ensure this, I found we do not configure maven-checkstyle-plugin  in oap pom.xml.


[Feature Request] Improvement Needed for Unit Tests

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
I recently reviewed the backward ops and found several bugs.

https://github.com/tenstorrent-metal/tt-metal/issues/6533
https://github.com/tenstorrent-metal/tt-metal/issues/6534
https://github.com/tenstorrent-metal/tt-metal/issues/6536
https://github.com/tenstorrent-metal/tt-metal/issues/6583
https://github.com/tenstorrent-metal/tt-metal/issues/6589
https://github.com/tenstorrent-metal/tt-metal/issues/6598

I believe there are two main reasons why there were so many bugs in backward ops:

The compare_result function often returns a pass even when the correct value and the TT result differ significantly. This may result in many bugs going unnoticed.
ex) https://github.com/tenstorrent-metal/tt-metal/issues/6598

The input data used in unit tests does not always reflect the characteristics of the ops
For instance, in the case of relu6, the gradient formula varies depending on whether the input falls within the range of 0 to 6. Therefore, to test all intervals effectively, the input data should include values around 0, 6, and nearby points, such as [-1, 0, 3, 6, 7].
However, currently, input data is generated using torch.randn, results in values mostly around [-1 to 1], neglecting testing in the vicinity of 6 and its surrounding intervals.

ex)

I didn't run all unit tests during the review, and only checked the suspicious parts, so I believe there are actually more bugs.
Improving unit tests seems to be a high priority to address recurring issues and find hidden bugs.
Describe the solution you'd like
A clear and concise description of what you want to happen.

Regarding the first issue, I'm not sure what the best solution would be.
For the second issue, a method is needed to input specific values related to the characteristics of the op as test data.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

[x] #6678
[x] #6676
[x] #6674
[ ] #6672
[x] #6589

yes. @razorback3. i will follow up and get back to you with a response. This is really not good.

@razorback3 @jliangTT @jvasilje We have updated the datagen and comparison function in PR #6679 for debugging.
We are parallel working on updating the test files with reference to this PR. Once the PR is merged with the main will submit the changes for all the Ops in a separate PR

@razorback3 @jliangTT @jvasilje We are also observing the following scenario (not sure whether this is an expected behavior),

if the input is filled with constant 1.0 we are getting nan as expected as a result
if the input is randomly filled for the same value 1.0 we are getting  a larger number as a result (as shown in the image) which results in a PCC failure issue

To handle this we have to add the condition separately in the logic. We are checking this kind of issue also.
Scenario 1:

Scenario 2:

[ ] #6678

Status:

@umadevimcw  and team has complete sweep of the existing test casest to close the gap in the test cases.
There are still issues that needs to be debugged but it is tracked separately.
the initial p0 sprint - to re-inspect the test quality is done and is recorded here
https://docs.google.com/spreadsheets/d/17vJReyuZgWtzlal_TSCff9v_lDrmLqNK/edit#gid=66577367

I think we can downgrade this to p1.

@jliangTT
Would you give access permission of google docs to me and @hschoi4448 ?
<EMAIL_ADDRESS><EMAIL_ADDRESS>

please see this doc - https://docs.google.com/spreadsheets/d/1VV-EwGJn1EgBN3jX3tg4TcX_yDkm5HAO/edit#gid=66577367
(sorry i have to made a copy due to not being able to get around access control)

will close this one for now. Can track the issue. Please re-open if you have any concerns.

OK. @hschoi4448 will double-check the result when he comes back from his vacation.

https://github.com/tenstorrent/tt-metal/issues/6583 still has a problem.

Test cdoe

# SPDX-FileCopyrightText: © 2023 Tenstorrent Inc.

# SPDX-License-Identifier: Apache-2.0

import torch
import pytest
import tt_lib
from tests.tt_eager.python_api_testing.unit_testing.backward_ops.utility_funcs import data_gen_pt_tt, compare_results

def data_gen_pt_tt(input_shapes, device, required_grad=False, val=1):
    pt_tensor = (torch.ones(input_shapes, requires_grad=required_grad) * val).bfloat16()
    tt_tensor = (
        tt_lib.tensor.Tensor(pt_tensor, tt_lib.tensor.DataType.BFLOAT16).to(tt_lib.tensor.Layout.TILE).to(device)
    )
    return pt_tensor, tt_tensor

@pytest.mark.parametrize(
    "input_shapes",
    (
        (torch.Size([1, 1, 32, 32])),
    ),
)
def test_bw_acosh(input_shapes, device):
    in_data, input_tensor = data_gen_pt_tt(input_shapes, device, True, val=0.5)
    grad_data, grad_tensor = data_gen_pt_tt(input_shapes, device, False, val=1)

    print("input_tensor", input_tensor)
    print("grad_tensor", grad_tensor)

    pyt_y = torch.acosh(in_data)

    tt_output_tensor_on_device = tt_lib.tensor.acosh_bw(grad_tensor, input_tensor)

    in_data.retain_grad()

    pyt_y.backward(gradient=grad_data)

    golden_tensor = [in_data.grad]

    comp_pass = compare_results(tt_output_tensor_on_device, golden_tensor)

    print("tt_output_tensor_on_device", tt_output_tensor_on_device)
    print("golden_tensor", golden_tensor)
    assert comp_pass

output

input_tensor ttnn.Tensor([[[[ 0.50000,  0.50000,  ...,  0.50000,  0.50000],
               [ 0.50000,  0.50000,  ...,  0.50000,  0.50000],
               ...,
               [ 0.50000,  0.50000,  ...,  0.50000,  0.50000],
               [ 0.50000,  0.50000,  ...,  0.50000,  0.50000]]]], shape=Shape([1, 1, 32, 32]), dtype=DataType::BFLOAT16, layout=Layout::TILE)
grad_tensor ttnn.Tensor([[[[ 1.00000,  1.00000,  ...,  1.00000,  1.00000],
               [ 1.00000,  1.00000,  ...,  1.00000,  1.00000],
               ...,
               [ 1.00000,  1.00000,  ...,  1.00000,  1.00000],
               [ 1.00000,  1.00000,  ...,  1.00000,  1.00000]]]], shape=Shape([1, 1, 32, 32]), dtype=DataType::BFLOAT16, layout=Layout::TILE)
2024-04-16 01:45:38.796 | ERROR    | tests.tt_eager.python_api_testing.sweep_tests.comparison_funcs:get_pcc:32 - One tensor is all nan, the other is not.
2024-04-16 01:45:38.796 | ERROR    | tests.tt_eager.python_api_testing.sweep_tests.comparison_funcs:get_pcc:32 - One tensor is all nan, the other is not.
2024-04-16 01:45:38.797 | DEBUG    | tests.tt_eager.python_api_testing.unit_testing.backward_ops.utility_funcs:compare_results:62 - False
2024-04-16 01:45:38.797 | DEBUG    | tests.tt_eager.python_api_testing.unit_testing.backward_ops.utility_funcs:compare_results:63 - False
2024-04-16 01:45:38.797 | DEBUG    | tests.tt_eager.python_api_testing.unit_testing.backward_ops.utility_funcs:compare_results:64 - Max ATOL Delta: nan, Max RTOL Delta: nan, PCC: 0.0, PCC check failed
tt_output_tensor_on_device [ttnn.Tensor([[[[inf     , inf     ,  ..., inf     , inf     ],
               [inf     , inf     ,  ..., inf     , inf     ],
               ...,
               [inf     , inf     ,  ..., inf     , inf     ],
               [inf     , inf     ,  ..., inf     , inf     ]]]], shape=Shape([1, 1, 32, 32]), dtype=DataType::BFLOAT16, layout=Layout::TILE)]
golden_tensor [tensor([[[[nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          ...,
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan],
          [nan, nan, nan,  ..., nan, nan, nan]]]], dtype=torch.bfloat16)]

Could you please check if the issues below are still problematic?

https://github.com/tenstorrent/tt-metal/issues/6583
https://github.com/tenstorrent/tt-metal/issues/6672
https://github.com/tenstorrent/tt-metal/issues/6737
https://github.com/tenstorrent/tt-metal/issues/6735
https://github.com/tenstorrent/tt-metal/issues/6728
https://github.com/tenstorrent/tt-metal/issues/6721
https://github.com/tenstorrent/tt-metal/issues/6720 : I saw a PR claiming to have resolved the issue https://github.com/tenstorrent/tt-metal/pull/7000. Why wasn't it merged?

@hschoi4448 for all the above tagged issues has hardware and performance limitations where handling/storing Nan /inf is the problem

In WHB0 storing Nan is possible
Storing Nan/inf in ckernel op is related to performance problem

If you check each issue we have added observation and even for few ops we have raised the PR with fix
PR is not merged due to pending approval from code owner

@hschoi4448 for all the above tagged issues has hardware and performance limitations where handling/storing Nan /inf is the problem
* In WHB0 storing Nan is  not possible

* Storing Nan/inf in ckernel op is related to performance problem

If you check each issue we have added observation and even for few ops we have raised the PR with fix
PR is not merged due to pending approval from code owner
Please find @rtawfik01 comment below
These checks also affect performance, are the users alright with a performance hit?
Also for tan op we can't support more range other than -1.45 to 1.45. For above this we have to do reduction operations with modulo operations which is not available.

Understood. If it's a hardware issue with limitations on performance and functionality, it seems that it's not something I can decide on, so I'll pass it on to my team. @razorback3

Could you please check if the issues below are still problematic?

https://github.com/tenstorrent/tt-metal/issues/6583 - In Progress
https://github.com/tenstorrent/tt-metal/issues/6672 - Assigned to TT https://github.com/tenstorrent/tt-metal/issues/14077
https://github.com/tenstorrent/tt-metal/issues/6735
https://github.com/tenstorrent/tt-metal/issues/6721 - Assigned to TT https://github.com/tenstorrent/tt-metal/issues/12776
https://github.com/tenstorrent/tt-metal/issues/6720 : PR Merged, Issue yet to close after discussion

Updates :

https://github.com/tenstorrent/tt-metal/issues/6583 - PR In Review
https://github.com/tenstorrent/tt-metal/issues/6672 - Assigned to TT https://github.com/tenstorrent/tt-metal/issues/14077
https://github.com/tenstorrent/tt-metal/issues/6735
https://github.com/tenstorrent/tt-metal/issues/6721 - Assigned to TT https://github.com/tenstorrent/tt-metal/issues/12776
https://github.com/tenstorrent/tt-metal/issues/6720 : Issue closed


2021.10-003 rockpi4c image doesn't boot - GPT  signature is wrong

I have tried booting the radxa-RockPi4C-2021.10-003 spi.installer.img off 2 different uSD cards on my rockpi4c. I don't see any errors when I try to boot it nor do I get any picture via micro HDMI. Over the serial port I get this:
DDR Version 1.20 20190314
In
channel 0
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
CS = 1
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 1
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
CS = 1
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 0 training pass!
channel 1 training pass!
change freq to 400MHz 0,1
channel 0
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
CS = 1
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 1
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
CS = 1
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 0 training pass!
channel 1 training pass!
change freq to 800MHz 1,0
Channel 0: LPDDR4,800MHz
Bus Width=32 Col=10 Bank=8 Row=15/15 CS=2 Die Bus-Width=16 Size=2048MB
Channel 1: LPDDR4,800MHz
Bus Width=32 Col=10 Bank=8 Row=15/15 CS=2 Die Bus-Width=16 Size=2048MB
256B stride
ch 0 ddrconfig = 0x101, ddrsize = 0x2020
ch 1 ddrconfig = 0x101, ddrsize = 0x2020
pmugrf_os_reg[2] = 0x3AA1FAA1, stride = 0xD
OUT
Boot1: 2018-06-26, version: 1.14
CPUId = 0x0
ChipType = 0x10, 221
Spi_ChipId = b4016
SpiBootInit:0
mmc0:cmd5,32
SdmmcInit=0 0
BootCapSize=0
UserCapSize=30436MB
FwPartOffset=2000 , 0
StorageInit ok = 151568
SecureMode = 0
SecureInit ret = 0, SecureMode = 0
GPT  signature is wrong
LoadTrust Addr:0x1800
LoadTrust Addr:0x1c00
LoadTrust Addr:0x2000
LoadTrust Addr:0x2400
LoadTrust Addr:0x2800
LoadTrust Addr:0x2c00
LoadTrust Addr:0x3000
LoadTrust Addr:0x3400
Addr:0x1800 No find trust.img!
LoadTrustBL error:-3
SecureMode = 0
SecureInit ret = 0, SecureMode = 0

Does this image boot for you @CRTified ? I flashed the uSD cards using balena etcher so it verifies them by default.

I did not have time to test the released image so far (The 4C runs my home automation and DNS, so it's hard to find a good time where it does not cause problems).
Just as a sidenote: On the matrix channel, there was another report about the image not booting.
The latest version I tested before the release was the one in this PR. But that one was self-built. Would be nice to know whether there were any problems.
As an alternative, you might want to try the version from the development-branch (where a lot of refactoring was done). I know of at least one user of an RockPi4A (OP1) that successfully uses the version on rev 1297c1bbb6ba882d5aa79f935fe1e55c243f639d, and the device trees of the 4A, 4B and 4C only differ marginally.
Sorry for the inconvenience. I'll try to see whether I have time to test it in the next few days.

Sounds like it might be best that I try building a dev image first, but how? I've looked at the README and the docs dir but I don't see any build instructions.
How do I build an image for the rockpi4c?

If you are under NixOS or have nix available on your distribution (which is AFAICT the only supported build method right now), it's a mere nix-build -A radxa-RockPi4C.
If I remember correctly, @samueldr was thinking about some other build options like a nix-based docker image, but I don't know about the progress there.

DDR Version 1.20 20190314
In
channel 0
[...]

This looks like the vendor RAM training blobs.
Note that if you have a U-Boot starting when no SD card is inserted, it will need to be bypassed in some way (usually device/vendor-dependent) to start the one on the SD card.

For bypassing, it should be possible by killing the SPI clock. In that case, have a look at the GPIO pinout and the SPI documentation.
Should be sufficient to tie pin 23 to GND (using a small resistor would be perfect, but shouldn't be necessary).
For building: You can use amd64 without any problems, and nix is available there (At this point, it's sufficient to look at its use for Tow-Boot as a glorified Makefile that tracks dependencies).
In case you trust me enough, you could also use the attached file (zipped as github won't allow me to upload *.img directly). I've built it on the previously mentioned revision of the development-branch (which is known-working for the 4A/B).
spi.installer.zip

I get the same result using the image in that zip.
I have tried putting a jumper over pins 23+35 but it doesn't seem to boot at all then.
Should I expect to see the towboot install menu via HDMI or is it something that requires a UART cable to view?

It should be visible on both UART and HDMI.

@gattohub I'm sure @CRTified will correct me but I don't think towboot supports the 4b unless it has a SPI chip.
I get the impression there were different revisions of he 4b and some don't have the SPI.
I'm going to be trying to build towboot dev later, I will update this thread after I've tried it.

I get the impression there were different revisions of the 4b and some don't have the SPI.

That is correct. According to the official docs, the SPI was not populated up to revision 1.3. But it's easy to solder in on your own (GD25Q127CSIG is relatively easy to source, the other chips mentioned there not so much).

I'm sure @CRTified will correct me if I'm wrong but I don't think towboot supports the 4b unless it has a SPI chip.

Tow-Boot should be able to be used in the place of a "classical" u-boot image on the eMMC or µSD-Card, but I did not test this.

Yes, the shared.disk-image.img is just that. It's pre-seeded with a GPT that covers the platform firmware partition with a protective partition for enhanced EBBR conformity.

I have builtt the development branch using the command @CRTified mentined above but its still not booting. I get no output via HDMI and this on the console, which seems to be the same as with the latest release build:
DDR Version 1.20 20190314
In
channel 0
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
CS = 1
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 1
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
CS = 1
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 0 training pass!
channel 1 training pass!
change freq to 400MHz 0,1
channel 0
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
CS = 1
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 1
CS = 0
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
CS = 1
MR0=0x18
MR4=0x1
MR5=0x1
MR8=0x8
MR12=0x72
MR14=0x72
MR18=0x0
MR19=0x0
MR24=0x8
MR25=0x0
channel 0 training pass!
channel 1 training pass!
change freq to 800MHz 1,0
Channel 0: LPDDR4,800MHz
Bus Width=32 Col=10 Bank=8 Row=15/15 CS=2 Die Bus-Width=16 Size=2048MB
Channel 1: LPDDR4,800MHz
Bus Width=32 Col=10 Bank=8 Row=15/15 CS=2 Die Bus-Width=16 Size=2048MB
256B stride
ch 0 ddrconfig = 0x101, ddrsize = 0x2020
ch 1 ddrconfig = 0x101, ddrsize = 0x2020
pmugrf_os_reg[2] = 0x3AA1FAA1, stride = 0xD
OUT
Boot1: 2018-06-26, version: 1.14
CPUId = 0x0
ChipType = 0x10, 221
Spi_ChipId = b4016
SpiBootInit:0
mmc0:cmd5,32
SdmmcInit=0 0
BootCapSize=0
UserCapSize=30436MB
FwPartOffset=2000 , 0
StorageInit ok = 151285
SecureMode = 0
SecureInit ret = 0, SecureMode = 0
GPT  signature is wrong
LoadTrust Addr:0x1800
LoadTrust Addr:0x1c00
LoadTrust Addr:0x2000
LoadTrust Addr:0x2400
LoadTrust Addr:0x2800
LoadTrust Addr:0x2c00
LoadTrust Addr:0x3000
LoadTrust Addr:0x3400
Addr:0x1800 No find trust.img!
LoadTrustBL error:-3
SecureMode = 0
SecureInit ret = 0, SecureMode = 0

I think I'll have to install Tow-Boot via the rpi4 mask rom case 2 procedure like this:
https://wiki.radxa.com/Rockpi4/dev/spi-install
I have got my rockpi4 to boot in maskrom mode but I've not been able to use rkdeveloptool to write it:
$ rkdeveloptool ld
DevNo=1	Vid=0x2207,Pid=0x330c,LocationID=101	Maskrom
# rkdeveloptool wl 0 /home/dan/spi.installer.img
Write LBA failed, err=13, can't open file: /home/dan/spi.installer.img

I installed rkdeveloptool via a snap under Ubuntu. I wonder if I build it from source or use a PPA to install rkdeveloptool if it'll work?

~/src/rkdeveloptool$ make
make[1]: Entering directory '/home/dan/src/rkdeveloptool'
g++ -DHAVE_CONFIG_H -I. -I./cfg  -Wall -Werror -Wextra -Wreturn-type -fno-strict-aliasing -D_FILE_OFFSET_BITS=64 -D_LARGE_FILE -I/usr/include/libusb-1.0   -g -O2 -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.cpp
main.cpp: In function ‘bool _Z9mergeBootv.part.0()’:
main.cpp:1493:36: error: ‘%s’ directive output may be truncated writing up to 557 bytes into a region of size 5 [-Werror=format-truncation=]
 1493 |  snprintf(buffer, sizeof(buffer), "%s", chip);
      |                                    ^~
......
 1534 |   chipType = convertChipType(chip + 2);
      |              ~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/stdio.h:867,
                 from DefineHeader.h:3,
                 from main.cpp:11:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:67:35: note: ‘__builtin_snprintf’ output between 1 and 558 bytes into a destination of size 5
   67 |   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
      |          ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   68 |        __bos (__s), __fmt, __va_arg_pack ());
      |        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
make[1]: *** [Makefile:489: main.o] Error 1
make[1]: Leaving directory '/home/dan/src/rkdeveloptool'
make: *** [Makefile:509: all-recursive] Error 1

well, i can confirm that with the armbiam uboot you get from the link i used above, you can boot from nvme with manjaro without sd.

@gattohub
Good news! It doesn't seem that Armbian ever released any install images for the 4c but I may be able to boot one of the archived 4b images or maybe build a 4c image with the build script and do what you've done?

@gattohub Just as a sidenote: The development branch also has an image for the 4A/B (they share their dtb).

There is a pull request to fix building rkdeveloptool here:
https://github.com/rockchip-linux/rkdeveloptool/pull/63
I still cannot flash towboot using the latest rkdeveloptool built from source:
./rkdeveloptool wl 0 /home/dan/spi.installer.img
Write LBA failed!

I will try using the Windows tool next.

I tried RKDevTool under Windows 10 and I got no further than under Linux. It can see my board is connected in maskrom mode but nothing else worked such as trying to erase to flash, download the bootloader or install the firmware.

I can confirm that instructions to install the armbian u-boot that @gattohub linked to also work on the 4c and I can now boot directly off my Samsung EVO 970 SSD. I used Armbian_21.08.1_Rockpi-4b_hirsute_edge_5.13.12_xfce_desktop.img to install the Armbian u-boot.
@gattohub - did you have to / could you successfully run:
echo spi1.0 > /sys/bus/spi/drivers/spi-nor/bind

I got an error about no such device existing after running that command but the rest of the instructions worked anyway. I will probably update the Rockpi4 wiki so I'd like to know if this command is required on the 4b?

./rkdeveloptool wl 0
/home/dan/spi.installer.img Write LBA failed!

This would have written the installer image, I believe to eMMC. I'm not sure rkdeveloptool has facilities to write to SPI.

Same problem here. 2021.10-004 rockpi4c image also does not boot.


Method for clearing the drawing

I have added a method clear to the drawing class that
removes all shapes with the exception of the background.
This is done by saving the background color of the image, than de-allocating
the array this shapes and re-adding the background.
Note: I am not sure whether this approach is method is completely memory-save,
as it acts on an array of polymorphic objects.
It is likely to be OK as log a s the "child" shapes  have a fixed size, but it might be insufficient
if newer ones (polylines ect. ) contain allocatable/pointer elements.
@perazz

Thank you.
Maybe  a "Null-Operation" would be useful.
I think that updating an existing draweing in non-interactive applications like video creation
(I do not think that video creation is far off the scope of this library. At the
end a video is just a series of images and while we will not write a MPEG codec
in pure Fortran in the near future, it is always possible to write a series of image
files or a stream of data in a pipe and let external tools like FFMPEG do the rest. )

Why did you close it?
I was about to merge it after fixing the conflict.


[dynamic regions] region is not split for hot write

Bug Report
When loading TPCC into 6 nodes, only three of them have heavy write flow. That's because hotspot is not split. We have develop bucket split in the past, what needs to do now is always enable bucket for dynamic regions.

/assign @bufferflies

/cc @bufferflies

yes, the hot region is too big and aggregate on one or two regions.  It needs to be splited by split-buckets  if the hot-region-schedule scheduled region is too big.


MSI B450 Tomahawk Max wrong color

Hi guys, thanks for this utility, on my motherboard MSI B450 Tomahawk Max it doesn't work well.
I'm trying the demo commands:
sudo ./target/release/msi-rgb 00000000 FFFFFFFF 00000000 # for green
and it give me color pusling from blue to violet
this:
sudo ./target/release/msi-rgb  FFFFFFFF 00000000 FFFFFFF
give me color pulsing from violet to green
and this:
sudo ./target/release/msi-rgb FFFFFFFF 00000000 00000000 -ib
give me color pulsing from green to yellow
If i can, i can help you to map the colors for this motherboard, just tell me what i have to do. Thanks

Can be closed as a duplicate of #106, since that has more discussion and context.


wemgehoert.de requires recapture?

While running a1fbox example2.py I noticed that the logs show lines like:
2020-11-29T10:35:50Z a1fbox-spamblock.blocker[5402]: WARNING:callinfo:404 Client Error: Not Found for url: https://www.wemgehoert.de/recaptcha/xxxxyyyyzzz

So it appears that the wemgehoert services does not/no longer work without a capture. Maybe it needs to be disabled by default? Looking at their website it seems they provide real API.

Can you provide the link to the real API? I only found this, which is somehow strange translated:
"Ist auch API zur Verfügung?
Die Programmierschnittstelle (API) ist von uns im Rahmen unserer Dienstleistung leider nicht zur Verfügung gestellt und für die Zukunft diese nicht geplant ist"

@bufemc Thank you! Sorry, what I wrote above was nonsense, I actually wanted to write "they provide no real API". Sorry for this.
I really like your project, it's super nice to be able to build a phone spam blocker from example2.py!
I did some work over the weekend to package it as a snap to deploy on a raspberry pi and also wrote a small seutp.py that I needed for the packaging. I hope to find time in the next few days to clean it up a little bit and push here as a PR.
Anyway, huge THANK YOU for this repo and your work on this.

It's running on a Raspi 4 here as well, see in README "Autostart on a Raspberry Pi". This was my lazy way.
It might be important to delay the start for some time, it seems the network (especially Wifi) might need some time until it can be used to reach out. Otherwise the call blocker will immediately terminate, maybe I should add a 30 sec waiting loop?
Small PRs are prefered instead of big ones, because if it's too big it really has to adapt/stick to the current coding style. Keep in mind the "API" is totally unfinished.. that's why it's INTENTIONALLY no package and I am not happy yet. Especially CallList is no module, and CallInfo and CallPrefix should be merged somehow IMHO. Big changes could follow.
WemGehoert is indeed a blocker, I already tried to simulate a browser with user agent, but seems this is not enough. Maybe should provide a "combination resolver" which does exclude Wemgehoert.

Thanks for these comments! I keep in mind that the API is not stable. And about the small PRs (but that will just happen by itself as I don't have much time unfortunately).
About the network, I put the example2.py into a systemd service as part of the snap so basic networking will be available via systemd dependencies and even if it dies systemd will just restart it again so eventually it will work. So I think it's fine to not do anything on the application level about this.
And API changes are's fine too, I am mostly interested in the project because of "example2.py" so as long as that keeps getting updated I'm fine :) I guess what I'm trying to say is that it's super useful for me as is already.

As can be seen above this issue has been adressed by now, so I may close it. Feel free to re-open it, if there should be any side effects.


Update keycloakVersion to 8.0.2 and get rid of propulsor

Update keycloakVersion to 8.0.2, and get rid of dep on propulsot. These will get rid of some vulnerabilities.

LGTM.


[Event Request] "OnAfterLookUp" for Requisition/Planning Worksheets Journal Names and the "Vendor No." on these Worksheets.

Could we get events called OnAfterLookUp of the journal names on both worksheets (they called "CurrentJnlBatchName" and "CurrentWkshBatchName")? We need that because the "OnAfterValidate" or the "OnAfterAfterLookup" Events are not triggered when using the Lookup of this Pagefields.
Requisition WorkSheet(CurrentJnlBatchName):
trigger OnLookup(var Text: Text): Boolean
begin
    CurrPage.SaveRecord;
    ReqJnlManagement.LookupName(CurrentJnlBatchName, Rec);
    CurrPage.Update(false);
    OnAfterLookUp(CurrentJnlBatchName, Rec);
end;

trigger OnValidate()
begin
    ReqJnlManagement.CheckName(CurrentJnlBatchName, Rec);
    CurrentJnlBatchNameOnAfterVali;
end;

[IntegrationEvent(false, false)]
local procedure OnAfterLookUp(var CurrentJnlBatchName: Code[10];var RequisitionLine: Record "Requisition Line")
begin
end;

Planning Worksheet (CurrentWkshBatchName):
trigger OnLookup(var Text: Text): Boolean
begin
    CurrPage.SaveRecord;
    ReqJnlManagement.LookupName(CurrentWkshBatchName, Rec);
    CurrPage.Update(false);
    OnAfterLookUp(CurrentWkshBatchName, Rec);
end;

trigger OnValidate()
begin
    ReqJnlManagement.CheckName(CurrentWkshBatchName, Rec);
    CurrentWkshBatchNameOnAfterVal;
end;

[IntegrationEvent(false, false)]
local procedure OnAfterLookUp(var CurrentWkshBatchName: Code[10];var RequisitionLine: Record "Requisition Line")
begin
end;

Similar to these we also need an OnAfterLookUp-Event for the "Vendor No." on the Pagefield. It only has an LookUp on table level.
It could look like this:
Requisition WorkSheet:
  field("Vendor No."; "Vendor No.")
  {
      ApplicationArea = Planning;
      ToolTip = 'Specifies the number of the vendor who will ship the items in the purchase order.';

      trigger OnLookup(var Text: Text): Boolean
      begin
          OnAfterLookUp(Rec);
      end;

      trigger OnValidate()
      begin
          ReqJnlManagement.GetDescriptionAndRcptName(Rec, Description2, BuyFromVendorName);
          ShowShortcutDimCode(ShortcutDimCode);
      end;
  }

[IntegrationEvent(false, false)]
local procedure OnAfterLookUp(var RequisitionLine: Record "Requisition Line")
begin
end;

Planning Worksheet:
  field("Vendor No."; "Vendor No.")
  {
      ApplicationArea = Planning;
      ToolTip = 'Specifies the number of the vendor who will ship the items in the purchase order.';
      Visible = false;

      trigger OnLookup(var Text: Text): Boolean
      begin
          OnAfterLookUp(Rec);
      end;
  }

[IntegrationEvent(false, false)]
local procedure OnAfterLookUp(var RequisitionLine: Record "Requisition Line")
begin
end;

The batch lookup events will be done.
The approach you suggested for vendor lookup triggers does not work. Please use existing events on table level.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


todo comments/commented out code in xIisModule.psm1

-Any updates on this 'bug' in both Get and Set TargetResource. can we delete this comment?
" # bug(TBD) deal with this better, maybe a seperate resource...."
-As well as this line of commented out code in Get-TargetResource:
"#-and $Module -ieq $handler.Module"

Probably, to be entirely honest I'm not 100% sure what that module is for...

@KarolKaczmarek Do you have any background on this? It looks like it was transferred over when this resource was moved to gitHub

I'm not the author, but the second comment can be removed.
Regarding the first one... it was created before open sourcing, you may want to follow up with @grayzu, @TravisEz13 or Arun (don't know his GitHub handle) since they have been originally working on xWebAdministration.


feat(styles): update tooltips position

Preview environments are getting posted here, as soon as they are ready!


remove unused connection attribute

fixes https://github.com/edgedb/edgedb-go/issues/126 which didn't have to do with options.
When running edgedb/edgedb-go against a development server (edb server) the server sent a ParameterStatus message that the packaged server builds don't seem to send. The bit of code that is changed in this PR is not usually exercised because the tests don't have a way of triggering server initiated messages.

Sorry @1st1 requesting a review was a reflex. Its probably not needed for this.

Sorry @1st1 requesting a review was a reflex. Its probably not needed for this.

NP ;)


Fix comment error

Fix comment error, please see the commit for the details.

Thanks for your contribution!


[#496] Distribute app without Firebase integration

Close https://github.com/nimblehq/android-templates/issues/496

What happened 👀
Remove Firebase integration from project
Insight 📝

Codemagic can distribute the app to Firebase App Distribution without integrating the Firebase to the project.
After merging this PR, the google-services.json in Github Secrets will be deleted.

Big thanks to @luongvo for noticing this and the POC branch. 🙇🏻♀
Proof Of Work 📹
Github Action to deploy on Firebase result:
https://github.com/nimblehq/android-templates/actions/runs/5712267786/job/15475425029
Codemagic to deploy on Firebase result:

All updated. @luongvo  🙏🏻

@luongvo I just recalled why I didn't remove it. If we use this github action workflow from .cicdtemplate in any project to distribute the app to the Firebase, that means we need to set up the Firebase to the project. So, we recommended saving the google-service.json file in the Github Secrets. 🤔  So IMHO, we don't need to remove that step to download the file in the .cicdtempleate folder. What do you think?

If we use this github action workflow from .cicdtemplate in any project to distribute the app to the Firebase, that means we need to set up the Firebase to the project.

Yes, we need to set up a new Firebase app on the Firebase console for distribution, but we do not need to add Firebase to the app. Hence, no google-services is required.

And we recommended saving the google-service.json file in the Github Secrets. 🤔 So IMHO, we don't need to remove that step to download the file in the .cicdtempleate folder.

We can keep that variable on GitHub Secrets for later usage. For now, we do not use it to execute any commands or to build the app, so I recommend removing the step to download it to avoid confusion.
@lydiasama Since you removed from both the root workflows and the .cicdtemplate, we should also remove it in https://github.com/nimblehq/android-templates/blob/3f234ab29313997961b4526956b877b3ee0e2a12/.github/workflows/review_pull_request.yml#L50-L58 🙏

@lydiasama @ryan-conway can we resolve and merge this PR asap? I'm working on KMM Templates and to reuse the Android app module, I need to have a successful build without this config 🙏


cifar10_cnn.py example fails to converge using rmsprop optimizer

The cifar10_cnn.py example as it is fails to converge. If rmsprop is changed to adam or replaced with an SGD(lr=0.01, momentum=0.9, decay=1e-6, nesterov=True) optimizer, the example works fine. Examples of the history for the 200 epochs in the example are shown below. Other than storing and dumping the results to file via cPickle to produce the plots, nothing else in cifar10_cnn.py was changed excepting the optimizers as stated.
Theano backend, GPU mode, Ubuntu 14.04, running on an AWS g2.2xlarge instance ("G2 Instances are backed by 1 x NVIDIA GRID GPU (Kepler GK104) and 8 x hardware hyperthreads from an Intel Xeon E5-2670").
(Also thanks for making Keras with such a great API.)

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.
Thank you!

[X] Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

[X] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

[X] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

[X] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Hi JefferyRPrice, may I know how you can save the performance metrics of train_loss, train_acc, val_loss, val_acc during the training process, and feed them into a plot. Would you like to share the script, directly related to this part? Thanks.

I modified cifar10_cnn.py that just imports cPickle, adds results = model.fit* to both fits (with and without augmentation), and then dumps results.history to a file. Here's the one that used SGD.
To plot, I used this code in an Jupyter Python 2 notebook.
For what it's worth, I did the experiment both with and without data augmentation with the same basic results, where rmsprop was wonky and adam and SGD were fine.

Thank you so much for sharing the code for dumping the result. I was also trying to generate the plot as you show. By the way, the metrics of loss corresponds to score, or cost, or objective ( as shown here at  https://keras.io/objectives/ ). But I was not able to find the definition of acc, e.g., accuracy. I can imagine what does it mean, but I want to know how does it defined. Do you happen to have any information on this? Thanks.

standard definition of classification accuracy I guess:
ACC = TP+TN/(P+N)
or for multiclass, just the trace of the confusion matrix C divided by number of samples:
np.trace(C)/np.sum(C)

Changing the rmsprop parameters to RMSprop(lr=0.0001, rho=0.95) results in convergence (see attached plot, which was generated with data_augmentation = False).
I would propose that rmsprop in the example be changed to adam so that the example works out-of-the-box.


Rename method

I think it is more readable, feel free to merge or close

yup, good one - extended it to the interface as well :+1:


Impossible to load local pretrained model for DPO Alignment

Hi,
I'm using Alignment-Handbook to finetune mistral on a custom dataset.
I've made the SFT Full Fine Tuning and it ended up well, and saved my pretrained model to ./data/zephyr-7b-sft-full.
Now i want to align my model with DPO, everything is okay with the loading of the dataset but it's impossible to load the pretrained model saved on my local machine.
Althought i've tried with absolute and relative path, it always throw this error :
/root/miniconda3/lib/python3.11/site-packages/datasets/table.py:1421: FutureWarning: promote has been superseded by mode='default'.
  table = cls._concat_blocks(blocks, axis=0)
Traceback (most recent call last):
  File "/workspace/work/finetuning/alignment-handbook/scripts/run_dpo.py", line 224, in <module>
    main()
  File "/workspace/work/finetuning/alignment-handbook/scripts/run_dpo.py", line 119, in main
    if is_adapter_model(model, model_args.model_revision):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/alignment/model_utils.py", line 99, in is_adapter_model
    repo_files = list_repo_files(model_name_or_path, revision=revision)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 110, in _inner_fn
    validate_repo_id(arg_value)
  File "/root/miniconda3/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 158, in validate_repo_id
    raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': './data/zephyr-7b-sft-full'. Use `repo_type` argument if needed.

and no matter how much I change, nothing happens.
Here is the /dpo/config_full.yaml file :
# Model arguments
model_name_or_path: ./data/zephyr-7b-sft-full

# Data training arguments
# For definitions, see: src/h4/training/config.py
# Data training arguments
dataset_mixer:
  /workspace/work/finetuning/tickets_dpo/: 1.0
dataset_splits:
- train
- test
preprocessing_num_workers: 12

# DPOTrainer arguments
bf16: true
beta: 0.1
do_eval: true
evaluation_strategy: steps
eval_steps: 100
gradient_accumulation_steps: 1
gradient_checkpointing: true
#hub_model_id: zephyr-7b-dpo-full
learning_rate: 5.0e-7
log_level: info
logging_steps: 10
lr_scheduler_type: linear
max_length: 1024
max_prompt_length: 1024
num_train_epochs: 5
optim: rmsprop
output_dir: data/zephyr-7b-dpo-full
per_device_train_batch_size: 8
per_device_eval_batch_size: 4
push_to_hub: false
save_strategy: "no"
save_total_limit: null
seed: 42
warmup_ratio: 0.1

Please help :)

Since you are using full fine-tuning, you can just skip the check for peft model by commenting the call of is_adapter_model .

@huchinlp yep I just noticed it and i dont skip this step but i've coded a list_local_files(model_path) function in order to not skip this step.
But yes it seems ok thanks 🙏