ERROR: No enum auxiliary type exists. At src/slow5.c:1458

Hi @Psy-Fer,
I am trying to convert some blow5 files to pod5 and get this error:
[slow5_get_aux_enum_labels::ERROR] No enum auxiliary type exists. At src/slow5.c:1458
06-Dec-23 14:45:55 - pyslow5 - [WARNING]: get_aux_enum_labels enum_labels is NULL
06-Dec-23 14:45:55 - pyslow5 - [WARNING]: get_header_value header value not found: ip_address - rg: 0
06-Dec-23 14:45:55 - pyslow5 - [WARNING]: get_header_value header value not found: local_bc_comp_model - rg: 0
06-Dec-23 14:45:55 - pyslow5 - [WARNING]: get_header_value header value not found: mac_address - rg: 0
Traceback (most recent call last):
  File "/srv/scratch/babsgenome/snakes/blow5/tiger/blue-crab-venv/bin/blue-crab", line 8, in <module>
    sys.exit(main())
  File "/srv/scratch/babsgenome/snakes/blow5/tiger/blue-crab-venv/lib/python3.10/site-packages/src/blue_crab.py", line 1529, in main
    slow52pod5(args)
  File "/srv/scratch/babsgenome/snakes/blow5/tiger/blue-crab-venv/lib/python3.10/site-packages/src/blue_crab.py", line 713, in slow52pod5
    s2s_s2p_worker(args, sfile, pod5_out)
  File "/srv/scratch/babsgenome/snakes/blow5/tiger/blue-crab-venv/lib/python3.10/site-packages/src/blue_crab.py", line 1161, in s2s_s2p_worker
    s5_end_reason = slow5_end_reason_labels[read.get("end_reason", 0)]
IndexError: list index out of range

Any ideas of what might cause this and how I might fix it?
Thanks!
Rich

Oh yea that's my bad.
Let me fix that and get back to you.

Hey,
Any chance you could show me the header columns of your data?
The first 2 lines above the actual reads (and below the header values)
It should be something like
#char*  uint32_t        double  double  double  double  uint64_t        int16_t*        enum{unknown,partial,mux_change,unblock_mux_change,data_service_unblock_mux_change,signal_positive,signal_negative}     char*   double  int32_t uint8_t uint64_t
#read_id        read_group      digitisation    offset  range   sampling_rate   len_raw_signal  raw_signal      end_reason      channel_number  median_before   read_number
     start_mux       start_time

What I'm looking for here is this part of it
enum{unknown,partial,mux_change,unblock_mux_change,data_service_unblock_mux_change,signal_positive,signal_negative}

This is the list that blue-crab tried to get from your slow5 file. If it's not present or fails, it tries to make it a list of just ["unknown"].
It looks like it's trying to use a value that is outside the length of the list. So having a look at the list is a good start to see if there is anything weird going on there.
An easy way to get that value from a blow5 file is to run this command
slow5tools view reads.blow5 | less

and just scroll down to that header line and copy paste it here.
Thanks
James

I have also just pushed a change to the dev branch that has a check on this line of code that will spit out what slow5_end_reason_labels is set to if it fails as a quick way to troubleshoot.
So another way is to switch to the dev branch, run pip install . again then try running the same conversion again and wait for it to hit the same error.
Thanks
James

Hi @Psy-Fer,
I am also getting the same error:
blue-crab s2p minion_sim_1000_itrs.blow5 -o minion_test.pod5
05-Apr-24 19:03:38 - blue-crab - [INFO]: single2single: 1 s/blow5 file detected as input. Writing 1:1 s/blow5->pod5 to file: minion_test.pod5
05-Apr-24 19:03:38 - blue-crab - [INFO]: Opening s/blow5 file: minion_sim_1000_itrs.blow5
[slow5_get_aux_enum_labels::ERROR] No enum auxiliary type exists. At src/slow5.c:1458
05-Apr-24 19:03:38 - pyslow5 - [WARNING]: get_aux_enum_labels enum_labels is NULL
Traceback (most recent call last):
File "/home/tomas/.local/bin/blue-crab", line 8, in
sys.exit(main())
File "/home/tomas/.local/lib/python3.10/site-packages/src/blue_crab.py", line 1529, in main
slow52pod5(args)
File "/home/tomas/.local/lib/python3.10/site-packages/src/blue_crab.py", line 713, in slow52pod5
s2s_s2p_worker(args, sfile, pod5_out)
File "/home/tomas/.local/lib/python3.10/site-packages/src/blue_crab.py", line 1161, in s2s_s2p_worker
s5_end_reason = slow5_end_reason_labels[read.get("end_reason", 0)]
IndexError: list index out of range
The last two lines before the actual reads are:
#char*	uint32_t	double	double	double	double	uint64_t	int16_tchar	double	int32_t	uint8_t	uint64_t
#read_id	read_group	digitisation	offset	range	sampling_rate	len_raw_signal	raw_signal	channel_number	median_before	read_number	start_mux	start_time
There is no enum{} in my files.
The slow5 files were generated (using the subprocess.run function of python) with the dna-r10-min model and full-contigs:
"squigulator " + fasta_derep + " -x dna-r10-min -o ./tmp/tmp_" + str(i) + ".slow5 --full-contig --seed " + str(random_numbers[i])
and then merged:
slow5tools merge tmp -o minion_sim_1000_itrs.slow5
The individual tmp files as well as the merged files have the same structure and no enum{} on line 9.
I tried the dna-r9-min model, and there also is no enum{} on line 9.
#char*	uint32_t	double	double	double	double	uint64_t	int16_tchar	double	int32_t	uint8_t	uint64_t
#read_id	read_group	digitisation	offset	range	sampling_rate	len_raw_signal	raw_signal	channel_number	median_before	read_number	start_mux	start_time
Thanks,
Tomas

Ahh so these reads were built with squigulator?
I'll need to tell @hasindu2008 to put a dummy end_reason in the blow5 output.
In the meantime, I'll modify blue-crab to insert a dummy enum via an argument, making all reads end in the signal_positive state.
I'll get back to you in a sec
James

Hi Tomas,
Could you please try using the dev branch and showing me the error it gives you?
You can do this by activating your environment
if you installed with pip from pypi, please clone the blue-crab repo
git clone git@github.com:Psy-Fer/blue-crab.git
Then go to the blue-crab repo and run git pull
then git switch dev
You can check it worked by running git status and it should say something like
On branch dev
Your branch is up to date with 'origin/dev'.

Then re-install this dev version into your env
pip install .
Now re-run your bluecrab command.
something fishy is going on, but this should figure it out.
Cheers,
James

Hi James,
Thanks for looking into this.
Here is the output of the dev branch:
blue-crab s2p minion_sim_1000_itrs.blow5 -o minion_test.pod5
10-Apr-24 09:04:21 - blue-crab - [INFO]: single2single: 1 s/blow5 file detected as input. Writing 1:1 s/blow5->pod5 to file: minion_test.pod5
10-Apr-24 09:04:21 - blue-crab - [INFO]: Opening s/blow5 file: minion_sim_1000_itrs.blow5
[slow5_get_aux_enum_labels::ERROR] No enum auxiliary type exists. At src/slow5.c:1458
10-Apr-24 09:04:21 - pyslow5 - [WARNING]: get_aux_enum_labels enum_labels is NULL
Traceback (most recent call last):
File "/home/tomas/.local/bin/blue-crab", line 8, in
sys.exit(main())
File "/home/tomas/.local/lib/python3.10/site-packages/src/blue_crab.py", line 1558, in main
slow52pod5(args)
File "/home/tomas/.local/lib/python3.10/site-packages/src/blue_crab.py", line 713, in slow52pod5
s2s_s2p_worker(args, sfile, pod5_out)
File "/home/tomas/.local/lib/python3.10/site-packages/src/blue_crab.py", line 1364, in s2s_s2p_worker
read_id=uuid.UUID(read["read_id"]),
File "/usr/lib/python3.10/uuid.py", line 177, in init
raise ValueError('badly formed hexadecimal UUID string')
ValueError: badly formed hexadecimal UUID string

Ahh progress!
Okay so now the issue is the readID isn't a valid uuid. Again I think that's a squigulator issue.
@hasindu2008 what are the readIDs you make?
The issue here is that pod5 requires the readID to be a uuid. So I can't just use any old string.
Ideally squigulator would create these and then blue-crab just reads the string and converts it.
Another option is in the absence of valid uuids I add an option to create one. But then you can't link the old reads to the new reads (unless I make a tsv file that provides the mapping).
What do you think?

James,
I agree, the solution is to have valid UUID and a dummy end_reason in the slow5/blow5 output generated by squigulator @hasindu2008.
Not having this also most likely breaks the butterfly-eel wrapper.
Ultimately, I need to be able to basecall the simulated slow5/blow5 files generated by squigulator so I can use the called fastq files for downstream analyses.
Thanks,
Tomas

Buttery-eel I can unbreak by using dummy uuids when inbasecall and then replace the original readID when the read comes back.
The issue is going over to pod5 you can't do this because of their strict typing. So yea, either squigulator produces uuids or I create them in blue-crab and give a file that maps squigulator readIDs with uuids.
Let's see what @hasindu2008 thinks and then we will implement it asap
James

Hey all,
The reason I adhere to the current readID format in squigulator is so that it is compatible with the "mapeval" utility in Minimap2's Paftools companion script. This is quite useful for assessing the mapping accuracy once the reads are basecalled. Also, I like deterministic read IDs compared to random ones.
It is very strange that POD5 needs the readid to be a UUID. Perhaps in their implementation, they simply store the UUID as a 128-bit integer instead of storing it as a variant-length string. This is not great, as this means POD5 is stuck with UUID forever as their read IDs, well, might change later and break backward compatibility.  ReadID in many bioinformatics formats including BAM format has been a variable string.
Perhaps, I can implement Squigulator an option called --ont-friendly  that produces some fake UUIDs for the read IDs, as well as a fake end_reason with the value "unknown".  Let me know your thoughts on this. This way, there is no need for the blue crab to do any "UUIdification" of the readIDs. If you all are happy, I can implement this to squigulator ASAP.
By the way, @Psy-Fer, is this UUID thing applicable to buttery-eel too? It wasn't a problem when using ont-guppy-server with the eel. Perhaps they enforced this UUID in ont-dorado-server? If they have enforced it (which is of limited sense to me), I would be very glad if you could do some internal mapping with a fake uuid when sending to the ont-basecall-server, but write the original readID to the FASTQ/SAM.

Also cross-referencing to the issue in squigulator that raises the same issue: https://github.com/hasindu2008/squigulator/issues/13

Hey,
Okay I'll just make absolutely sure what pod5 is doing so we are 100% correct when we do this.
James

@hasindu2008 and @Psy-Fer
Perhaps, I (@hasindu2008) can implement Squigulator an option called --ont-friendly that produces some fake UUIDs for the read IDs, as well as a fake end_reason with the value "unknown". Let me know your thoughts on this. This way, there is no need for the blue crab to do any "UUIdification" of the readIDs. If you all are happy, I can implement this to squigulator ASAP.
I think this is a great solution that will maintain maximum compatibility for downstream use.
Thanks,
Tomas

Okay I have confirmed that pod5 requires a uuid type for the readID, even though it shouldn't have to be.
--s2p--
verbose=1
-------------------blue-crab version-------------------
SLOW5/BLOW5 <-> POD5 converter version: 0.1.0

-------------------testcase:1: .slow5 to .pod5-------------------
12-Apr-24 17:35:30 - blue-crab - [INFO]: single2single: 1 s/blow5 file detected as input. Writing 1:1 s/blow5->pod5 to file: ./test//data/out/s2p/a.pod5
12-Apr-24 17:35:30 - blue-crab - [INFO]: Opening s/blow5 file: ./test//data/raw/s2p/a.slow5
12-Apr-24 17:35:30 - pyslow5 - [WARNING]: get_header_value header value not found: ip_address - rg: 0
12-Apr-24 17:35:30 - pyslow5 - [WARNING]: get_header_value header value not found: mac_address - rg: 0
Traceback (most recent call last):
  File "/home/jamfer/pvenv/blue-crab-test/bin/blue-crab", line 8, in <module>
    sys.exit(main())
  File "/home/jamfer/pvenv/blue-crab-test/lib/python3.8/site-packages/src/blue_crab.py", line 1561, in main
    slow52pod5(args)
  File "/home/jamfer/pvenv/blue-crab-test/lib/python3.8/site-packages/src/blue_crab.py", line 713, in slow52pod5
    s2s_s2p_worker(args, sfile, pod5_out)
  File "/home/jamfer/pvenv/blue-crab-test/lib/python3.8/site-packages/src/blue_crab.py", line 1392, in s2s_s2p_worker
    writer.add_read(read)
  File "/home/jamfer/pvenv/blue-crab-test/lib/python3.8/site-packages/pod5/writer.py", line 256, in add_read
    self.add_reads([read])
  File "/home/jamfer/pvenv/blue-crab-test/lib/python3.8/site-packages/pod5/writer.py", line 292, in add_reads
    *self._prepare_add_reads_args(reads),
  File "/home/jamfer/pvenv/blue-crab-test/lib/python3.8/site-packages/pod5/writer.py", line 306, in _prepare_add_reads_args
    [np.frombuffer(read.read_id.bytes, dtype=np.uint8) for read in reads]
  File "/home/jamfer/pvenv/blue-crab-test/lib/python3.8/site-packages/pod5/writer.py", line 306, in <listcomp>
    [np.frombuffer(read.read_id.bytes, dtype=np.uint8) for read in reads]
AttributeError: 'str' object has no attribute 'bytes'
testcase 1 failed

This is what happens if we just parse a str
it's trying to access the bytes method on the uuid type specifically, as that is what they expect.
So yea, I think we need to go with dummy uuids, and just make a tsv file that maps the uuid with the more verbose read information you want to store.
James

@Psy-Fer I am implementing an option in squigulator to generate uuids for readids, so blue-crab does not need to do anything.
Please check if the buttery-eel is also broken due to  this uuid thing?

I should probably merge the buttery-eel/skipped branch into main and do a release to handle this.

@killidude
If you compile squigulator from the dev branch, and specify the option --ont-friendly=yes it should be pod5 conversion compatible.
When you specify --ont-friendly=yes it will add a dummy end_reason and create fake UUID for read IDs so.
If you encounter issues let me know, thanks.
Seems like buttery-eel works even without things being uuid as James mentioned above.

James,
I agree, the solution is to have valid UUID and a dummy end_reason in the slow5/blow5 output generated by squigulator @hasindu2008.
Not having this also most likely breaks the butterfly-eel wrapper.
Ultimately, I need to be able to basecall the simulated slow5/blow5 files generated by squigulator so I can use the called fastq files for downstream analyses.
Thanks,
Tomas

@hasindu2008,
Thanks for implementing this option. I can now convert the squigulator generated files to pod5.
Thanks for your help,
Tomas

Hi @Psy-Fer ,
I'm using squigulator (v0.4.0) with the --ont-friendly=yes parameter and blue-crab (v0.2.0):
The error occurs during the conversion of a squigulator .slow5 file to .pod5. Here’s the error traceback:
04-Oct-24 16:12:25 - blue-crab - [INFO]: single2single: 1 s/blow5 file detected as input. Writing 1:1 s/blow5->pod5 to file: test.pod5
04-Oct-24 16:12:25 - blue-crab - [INFO]: Opening s/blow5 file: squigulator_reads.slow5
Traceback (most recent call last):
  File "/X.local/bin/blue-crab", line 8, in <module>
    sys.exit(main())
  File "/X/.local/lib/python3.8/site-packages/src/blue_crab.py", line 1562, in main
    slow52pod5(args)
  File "/X/.local/lib/python3.8/site-packages/src/blue_crab.py", line 717, in slow52pod5
    s2s_s2p_worker(args, sfile, pod5_out)
  File "/X/.local/lib/python3.8/site-packages/src/blue_crab.py", line 1195, in s2s_s2p_worker
    reason, forced = s2p_end_reason_convert(s5_end_reason)
  File "/X/.local/lib/python3.8/site-packages/src/blue_crab.py", line 94, in s2p_end_reason_convert
    "api_request": (p5.EndReasonEnum.API_REQUEST, False),
  File "/usr/lib/python3.8/enum.py", line 384, in __getattr__
    raise AttributeError(name) from None
AttributeError: API_REQUEST

I suspect this issue might be related to a recent pull request  based on the new pod5 spec from about a month ago. Is there a way to avoid this error?

Hmm..make sure you have the latest pod5 version?
Which version do you have? Please do a pip list for me?

Thank you for the fast answer, upgrading pod5 fixed the problem!


Better & cheaper initial experience on Heroku

This PR does a few things to make the process of setting up an app on Heroku better and cheaper.

It adds a buildpack for jemalloc and sets the JEMALLOC_ENABLED ENV var. This helps with memory usage. Fixes: https://github.com/bullet-train-co/bullet_train/issues/1524
Makes the initial setup cheaper by using cheaper dynos and the entry level for a number of addons (in addition to removing the Bucketeer addon that seems questionably useful to begin with). This brings the monthly cost (as provisioned by app.json) down from ~$185/month to ~$22/month. Fixes: https://github.com/bullet-train-co/bullet_train/issues/1127
Removes the Sentry addon. This was redundant since Honeybadger was already included, and Sentry isn't supported in review apps on Heroku, so having it in app.json prevents the use of review apps entirely which is frustrating. Fixes: https://github.com/bullet-train-co/bullet_train/issues/1521
Sets RAILS_MAX_THREADS and WEB_CONCURRENCY to match the values suggested by heroku. Fixes: https://github.com/bullet-train-co/bullet_train/issues/1523

Joint PR for udpating docs: https://github.com/bullet-train-co/bullet_train-core/pull/850

We could potentially take another $7 off the monthly cost by defaulting to embedded Sidekiq instead of having a worker dyno.

We could potentially take another $7 off the monthly cost by defaulting to embedded Sidekiq instead of having a worker dyno.

This definitely sounds interesting too! I don't know how embedded is to run though since I'm not much of an Ops person.


default local not working

hi
i use set_locale 2.4.0 with gettext 0.13.1 but default_locale in config or router not working and when i use localhost:4000  just showing en but my default_locale is ru
this is my source https://github.com/shahryarjb/ESOGTIPH

i wrote a plug which   changed accept_lange Before conn  being sent to set_locale plug
my problem is solved
def call(conn, _params) do
conn =  conn  |> put_req_header("accept-language", "ru")
conn
end
but one question why cookie or config not work ?

i wrote a plug which   changed accept_lange Before conn  being sent to set_locale plug
my problem is solved
def call(conn, _params) do
conn =  conn  |> put_req_header("accept-language", "ru")
conn
end
but one question why cookie or config not work ?


Profiler does not work correctly on Linux or Win64

On linux and 64bit windows, the profiler timing is incorrect.
Profiles on these platforms will contain percentage time values < 0 and > 100%. This can be seen in the enterprise profiler dump which was generated on 64 bit windows.
The reason is that the profiler timing ASM (in profiler.cpp) is only compiled when TORQUE_SUPPORTS_VC_INLINE_X86_ASM or TORQUE_SUPPORTS_GCC_INLINE_X86_ASM are defined.
TORQUE_SUPPORTS_VC_INLINE_X86_ASM and TORQUE_SUPPORTS_GCC_INLINE_X86_ASM are not defined for  LINUX or WIN64.

Ahhhhh that explains it. Wow, okay. Should we consider moving this out of ASM?

Keeping this open as a reminder to sort the high-res timer out.

add an option to use the SDL high performance counter. works great and is cross platform.

Moving to 3.9 for now. We'll shift to the SDL counter when using SDL then.

:+1:


BPF file format doesn't match what expected of revcor project

The revcor repo offers the possibility to quickly build reverse correlation experiments with files generated by cleese. However, it expect bpf files to

have exactly the same name as the stimulus file, with a txt extension (ex. sound001.wav and sound001.txt)
that columns of the bpf are labeled with a header and comma separated (ex. time, pitch\n 0, 150)

That format isn't enforced in how PhaseVocoder saves bpf files in batch mode (default right now is sound001_bpf.txt, no header and Tab-separated values). This should be fixed either hard in code, or via the config file (see e.g. user-specified extensions for Mediapipe dfm)

[update] on it. have modified the file extension, and FaceWarp header. Need to add header in PhaseVocoder.


Is FirstData_Connect really supported?

Based on the following snippets, it seems to me like it isn't:
class CompletePurchaseRequest extends PurchaseRequest
{
    public function sendData($data)
    {
        return $this->response = new CompletePurchaseResponse($this, $data);
    }

class PurchaseResponse extends AbstractResponse implements RedirectResponseInterface
{
    public function isSuccessful()
    {
        // FIXME: This makes no sense.
        return false;
    }

As far as I can see, the FirstData_Connect gateway never issues a post request. It just creates a response object using the request data.
Am I missing something? Otherwise it would be good to remove it from the readme.

If a response is always going to be a redirect, then it will return false for isSuccessful() - it does make sense.
isSuccessful() only returns true if the payment is accepted and totally finished. If there are still steps to do - a redirect in this case - them it will return false. It historically has come from the very early days of Omnipay when all payments were done via API and no redirects were involved, if that helps it to make more sense.

I'll reopen this because I'm now confused by something else. Hopefully someone can clarify if I'm missing something. All the credit card information is required before redirecting:
class PurchaseRequest extends AbstractRequest
{
    public function getData()
    {
        $this->validate('amount', 'card');
         ...
         // If no hosted data, or a number is passed, validate the whole card
        if (is_null($this->getHostedDataId()) || ! is_null($this->getCard()->getNumber())) {
            $this->getCard()->validate();
        } elseif (is_null($this->getCard()->getCvv())) {
            // Else we only require the cvv when using hosted data
            throw new InvalidCreditCardException("The CVV parameter is required when using hosteddataid");
        }

        $data['cardnumber'] = $this->getCard()->getNumber();
        $data['cvm']        = $this->getCard()->getCvv();
        $data['expmonth']   = $this->getCard()->getExpiryDate('m');
        $data['expyear']    = $this->getCard()->getExpiryDate('y');

        $data['bname']    = $this->getCard()->getBillingName();
        $data['baddr1']   = $this->getCard()->getBillingAddress1();
        ...

doesn't this defeat the purpose of using a redirect? i.e: not passing credit card data through my servers?

It requires cc data because the option full_bypass is set to true.

@acarpio89 Hi, can you please show me in which lines of the code is the validation of credit card disabled based on full_bypass?

@barryvdh


Update COPY parameter for Parquet writing, remove ALLOW_OVERWRITE and add OVERWRITE_OR_IGNORE

As stated in the discord chat:

I tried to write a Parquet file via a COPY statement with ALLOW_OVERWRITE TRUE. This yields an error: Not implemented Error: Unrecognized option for PARQUET: ALLOW_OVERWRITE I'm pretty sure this worked with past versions. Shall I create an issue, or is this by design? It's included in the docs at https://duckdb.org/docs/extensions/httpfs#writing

If I omit it, I get an Error: IO Error: Directory s3:mybucket/stats is not empty! Enable OVERWRITE_OR_IGNORE option to force writing error message... This OVERWRITE_OR_IGNORE isn't included in the docs, at least I don't find it.

cc @Mause

Thanks!


DG1: gpu hangs on multi-process decoding w/ RAM constrains

I run simple_2_decode_vmem to decode 1080p video on the DG1 platform with 8GB video memory. when I run it in 120 processes it will be blocked at somewhere, but when I run it in 100 processes it works well and will be finished in 5 minutes. If I force it to allocate 32 surfaces (set Request.NumFrameSuggested to 32 before it call alloc() ) and run simple_2_decode_vmem in 50 processes, some processes will be blocked and some will report GPU_HANG error when calling DecodeFrameAsync().
I can also produce this issue on sample_decode (command line: sample_decode h264 -device /dev/dri/renderD128 -vaapi -hw -i video_1920x1080.264 -o /dev/null ). After several testing I found that the error only occur when I run program in multi-process and the video memory used is over 4GB and only on the platform with 8GB video memory (DG1 with 4GB works fine).

DG1 is a new platform. Not all the patches has landed to Linux kernel (even to drm-tip). Thus, you need to be extremely careful with system setup. @wenbinc-Bin : can you, please, clarify which kernel you have used? In particular, currently supported system configuration is described here: https://dgpu-docs.intel.com/devices/iris-xe-max-graphics/index.html. Can you, please, check/confirm whether you use kernel for DG1 as prescribed by this article? If not - please, try to follow the guide at least concerning DG1 kernel setup.

DG1 is a new platform. Not all the patches has landed to Linux kernel (even to drm-tip). Thus, you need to be extremely careful with system setup. @wenbinc-Bin : can you, please, clarify which kernel you have used? In particular, currently supported system configuration is described here: https://dgpu-docs.intel.com/devices/iris-xe-max-graphics/index.html. Can you, please, check/confirm whether you use kernel for DG1 as prescribed by this article? If not - please, try to follow the guide at least concerning DG1 kernel setup.

@dvrogozh I see the link  (https://dgpu-docs.intel.com/devices/iris-xe-max-graphics/index.html) is about dGPU Virtualization. My machine is desktop and iGPU is disabled. Here is my environment:
kernel:       Linux 5.4.48+dg1140
drm:          10dd3eb6d5ab6c1867e1ca
libva:         e0f6e56fde81e2aa8724d
libva-util:   83728b910dc72cad5750
xf86:          a511f22cdec5650491
gmmlib:       6f15b795e1511febf233b4
media-driver: 80dd4ffdb6cd4ff8885
msdk:             5b9efad9acf462af
lspci | grep VGA
03:00.0 VGA compatible controller: Intel Corporation Device 4905 (rev 01)
I think the kernel I am using supports DG1 and I don't need dGPU Virtualization. Do I need to completely follow the guide to set up the VM environment?

@dvrogozh I see the link  (https://dgpu-docs.intel.com/devices/iris-xe-max-graphics/index.html) is about dGPU Virtualization. My machine is desktop and iGPU is disabled. Here is my environment:
kernel:       Linux 5.4.48+dg1140
drm:          10dd3eb6d5ab6c1867e1ca
libva:         e0f6e56fde81e2aa8724d
libva-util:   83728b910dc72cad5750
xf86:          a511f22cdec5650491
gmmlib:       6f15b795e1511febf233b4
media-driver: 80dd4ffdb6cd4ff8885
msdk:             5b9efad9acf462af
lspci | grep VGA
03:00.0 VGA compatible controller: Intel Corporation Device 4905 (rev 01)
I think the kernel I am using supports DG1 and I don't need dGPU Virtualization. Do I need to completely follow the guide to set up the VM environment?

@dvrogozh I update the kernel to dg1148, and the problem is partially resolved. Now I can run sample_decode in multi-process and use device memory over 4GB, but if I try to use memory over 8GB, which will be out of memory, it still reports GPU HANG error.
dmesg output:
[68756.688810] i915 0000:03:00.0: [drm] GPU HANG: ecode 12:0:00000000
[68756.688824] i915 0000:03:00.0: [drm] Resetting bcs0 for stopped heartbeat on bcs0
[68757.359583] i915 0000:03:00.0: [drm] Resetting bcs0 for preemption time out
[68757.360567] i915 0000:03:00.0: [drm] GPU HANG: ecode 12:2:18800102, in  [0]
[68771.792799] i915 0000:03:00.0: [drm] GPU HANG: ecode 12:0:00000000
[68771.792813] i915 0000:03:00.0: [drm] Resetting bcs0 for stopped heartbeat on bcs0

@dvrogozh I update the kernel to dg1148, and the problem is partially resolved. Now I can run sample_decode in multi-process and use device memory over 4GB, but if I try to use memory over 8GB, which will be out of memory, it still reports GPU HANG error.
dmesg output:
[68756.688810] i915 0000:03:00.0: [drm] GPU HANG: ecode 12:0:00000000
[68756.688824] i915 0000:03:00.0: [drm] Resetting bcs0 for stopped heartbeat on bcs0
[68757.359583] i915 0000:03:00.0: [drm] Resetting bcs0 for preemption time out
[68757.360567] i915 0000:03:00.0: [drm] GPU HANG: ecode 12:2:18800102, in  [0]
[68771.792799] i915 0000:03:00.0: [drm] GPU HANG: ecode 12:0:00000000
[68771.792813] i915 0000:03:00.0: [drm] Resetting bcs0 for stopped heartbeat on bcs0

Sorry I made the mistake. Updating the kernel cannot fix problem. The issue is still there.

Sorry I made the mistake. Updating the kernel cannot fix problem. The issue is still there.


Update to v1.1.0

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-webservice.
I tried to re-render for you, but it looks like there was nothing to do.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Failing because r-httpuv >=1.4.3 is unavailable from conda-forge.

@conda-forge-admin, please rerender

Depends on https://github.com/conda-forge/r-sourcetools-feedstock/pull/3

Depends on adding R 3.5.1 builds to these:

https://github.com/conda-forge/r-httpuv-feedstock/pull/9
https://github.com/conda-forge/r-htmltools-feedstock/pull/2
https://github.com/conda-forge/r-xtable-feedstock/pull/1

@bgruening This is now good to merge 🎊

:tada:


.NET Core attach debugger template defaults to "processname" and the flydown says use "pid"

When you try to debug attach a .NET Core application the default launch.json has "processname", but if you use "dotnet" it says it has found multiple and you need to use "pid".  But in reality you need to use "processId".  I recommend that we change the default template to replace "processName" with "processId" and update the fly down text to use "processId" instead of "pid".

Sounds reasonable. @caslan @DustinCampbell anyone have an opinion as to if we should change this for RC2?

It'd be nice to make it work. :smile:

It's very low risk so I'm in favor of changing especially now running apps via dotnet ... is more common

Yes, should be ~0 risk.

@MattGertz are you okay taking this?

I'm OK but it will need an escrow check from Dmitry/JoC.

Talked with Gregg offline; I didn't realize that there was a distinct part from the error message change.  This won't meet an escrow bar; we can pick it up afterwards.

Sounds good. Anyway, there'll be interim releases available that we can point customers to afterward.

Fixed with https://github.com/OmniSharp/omnisharp-vscode/pull/197

Was this issue fixed?
I tried the v1.0.5-rc2, it still stays there.

@kerryjiiang: It is fixed in the dev branch in our RTM milestone.


TestCafe native CLI params

Is that possible to pass testcafe's native params while running codeceptjs tests?
I mean I cant start testcafe's tests like this:

testcafe -c 4 safari,firefox tests/test.js

Is that somehow possible to pass that params when I do this:

npx codeceptjs run ...

Sorry, we don't support running in multiple browsers at once, as well as setting a browser from CLI. You can do this from the configuration.

@DavertMik why is that? For example, I'd like to pass teamcity reporter as I need integrate with teamcity, but there is hardcoded 'minimal' reporter as testcafe initied through your code and codeceptjs has no integratioin module with TeamCity, but testcafe itself has, so that would be nice to pass CLI params through helpers config, isnt that option?


multiplatform support

Hi @kropp,
I updated the sample project to kotlin multiplatform and changed it so it depends on the gtk3 project.
PlacesSidebar.onOpenLocation is missing and commented out in fileManager.kt.
I moved the generated code to kotlin to generatedKotlin.
There are also two bugfixes that are required to build and run the sample project (radioButton and scrolledWindow have to accept null).
You can now run:
cd gtk3
./gradlew publishToMavenLocal
cd ../sample
./gradlew build

I still need to change the generator, but I send you a PR so you can drop me a note if you want.

Wow! Impressive!
Thanks a lot for your contribution. I need some time to go through it.

It would be super duper awesome if this could be released with KN 1.4 support

I am curious is this project still maintained?

Would love to see this merged as well. This is the most promising project I've seen so far for a kotlin-native desktop GUI.

Hi everyone, since this project seems to have lost its momentum, I was wondering if anyone would be interested in contributing to this other project that provides already some basic support for GTK4 and source code generation from GIR files: https://gitlab.com/gtk-kn/gtk-kn
Some documentation is available here: https://gtk-kn.org/


fix the CTC zipformer2 training

too many supervision tokens
change filtering rule to if (T - 2) < len(tokens): return False
this prevents inf. from appearing in the CTC loss value (empirically tested)

workflow: https://github.com/k2-fsa/icefall/actions/runs/10348851808/job/28642009312?pr=1713
fatal: unable to access 'https://huggingface.co/Zengwei/icefall-asr-librispeech-zipformer-2023-05-15/': Recv failure: Connection reset by peer

Hi @csukuangfj ,
how about this one ? Is @yaozengwei testing it currently ?
It is solving the issue https://github.com/k2-fsa/icefall/issues/1352
My theory is that CTC uses 2 extra symbols at beginning/end of label sequence.
So, the label-length limit needs to be lowered by 2 symbols to accomodate that.
Best regards
Karel

Sorry for the late reply.
Could you analyze the wave that causes inf loss?
Is it too short?
Does it contain only a single word or does it contain nothing at all?

Hi,
the problematic utterance contained many words:
(num_embeddings, supervision_length, difference a-b) = (34, 33, 1)
text:
['▁O', 'f', '▁all', '▁.', '▁P', 'ar', 'li', 'a', 'ment', '▁,', '▁Co', 'un', 'c', 'il', '▁and', '▁Co', 'm', 'm', 'i', 's', 's', 'ion', '▁are', '▁work', 'ing', '▁to', 'ge', 'ther', '▁to', '▁de', 'li', 'ver', '▁.']
It seems like a better set of BPEs could reduce the number of supervision tokens.
Nevertheless, this would only hide the ``inf.'' problem for CTC.
I believe the two extra tokens for the CTC loss are the <bos/eos>
that get (pre-,ap-)pended to the supervision sequence,
hence the (T - 2).
Best regards
Karel

the problematic utterance contained many words:

Thanks for sharing! Could you also post the duration of the corresponding wave file?

This is the corresponding Cut:
MonoCut(id='20180612-0900-PLENARY-3-59', start=557.34, duration=1.44, channel=0, supervisions=[SupervisionSegment(id='20180612-0900-PLENARY-3-59', recording_id='20180612-0900-PLENARY-3', start=0.0, duration=1.44, channel=0, text='Of all . Parliament , Council and Commission are working together to deliver .', language='en', speaker='None', gender='male', custom={'orig_text': 'of all. Parliament, Council and Commission are working together to deliver.'}, alignment=None)], features=Features(type='kaldi-fbank', num_frames=144, num_features=80, frame_shift=0.01, sampling_rate=16000, start=557.34, duration=1.44, storage_type='lilcom_chunky', storage_path='data/fbank/voxpopuli-asr-en-train_feats/feats-59.lca', storage_key='395124474,12987', recording_id='None', channels=0), recording=Recording(id='20180612-0900-PLENARY-3', sources=[AudioSource(type='file', channels=[0], source='/mnt/matylda6/szoke/EU-ASR/DATA/voxpopuli/raw_audios/en/2018/20180612-0900-PLENARY-3_en.ogg')], sampling_rate=16000, num_samples=139896326, duration=8743.520375, channel_ids=[0], transforms=None), custom={'dataloading_info': {'rank': 3, 'world_size': 4, 'worker_id': None}})
It is a 1.44 sec long cut in a very long recording (2.42 hrs).
And the 1.44 sec is very little to pronounce all the words in the reference text :
"Of all . Parliament , Council and Commission are working together to deliver ."

yes, I think it should be good to filter out such kind of data.

Hello, is there something needed for this to merge from my side ?
K.

Well, without that (T - 2) change i was getting inf. value from the CTC loss.
There sholud be no inf. even if the data are prepared badly.
I also did not find any trace of the extra CTC symbols  or  or similar in the scripts.
The torch.nn.functional.ctc_loss(.) is getting the same set of symbols as transducer loss.
Could you try to reproduce the issue by adding a training example with a very lenghty transcript ?
(or I can create a branch to demonstrate it, say repeating the librispeech transcript 100x, just to make the error appear)
Best regards,
Karel


hotfix/parking -> master

This change is

Review status: 0 of 7 files reviewed at latest revision, all discussions resolved.

a discussion (no related file):
Align Add/Edit Vehicle cell texts

a discussion (no related file):
Change title of Add/Edit dynamically

Comments from Reviewable

Reviewed 1 of 4 files at r4, 1 of 2 files at r5.
Review status: 1 of 7 files reviewed at latest revision, 2 unresolved discussions.

a discussion (no related file):
Broken info buttons

a discussion (no related file):
Update text on iOS 9/10

Comments from Reviewable

Reviewed 2 of 6 files at r3, 2 of 4 files at r4, 1 of 2 files at r5, 3 of 3 files at r6, 1 of 1 files at r7.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable

Reviewed 1 of 1 files at r8.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Extract router to a gem?

It's awesome that Grape was able to switch off of Rack::Mount to a in-tree router solution! However, many other Ruby web frameworks (ex: Hanami/Lotus) are also still using fairly unmaintained libraries like http_router among others. Does it make sense to extract Grape's router to a gem that can be used and shared by all? Does it make sense to refactor Grape's router further to use Rails' router, Journey?
cc @dblock

That's basically https://github.com/namusyaka/pendragon, isn't it? /cc: @namusyaka.
Journey has been merged into Rails. The last thing we want is to add a Rails dependency, that'd be awkward :)

Yes, The concept of pendragon is same with new router, but new router is optimized for Grape.
If you wish to use grape router in other projects, I'm going to update and refactor pendragon.
Thanks!

I'm going to close this, it's beyond Grape's scope.


Issue with DB username/pwd

YAML file has MongoDB username and password visible, which is not a good practice. Can you use environment variable for this.

We will add .env files for this issue

Fixed to use dotenv
https://github.com/Bala-Logesh/SE23-61-Application-Tracking-System/commit/3646ca00f8b7519eefb81d4af471bef3016d3943


hide stacktrace error during driver init for non final attempt

we do produce huge error+stacktrace for every driver start retry.
let's optimize a little bit and print only error if it is not final init_retry_count for driver.
And print error+stacktrace only for last attempt

done. it might be tested using 7.3.17-SNAPSHOT carina with overridden properties:
selenium_url=http:///something-invalie/wd/hub
init_retry_count=2
init_retry_interval=0

after launching any web test with above settings we should see 2 error messages without stacktrace and 3rd one with huge stacktrace with details:
Driver initialization 'default' FAILED! Retry 1 of 3 time - unable to achieve/connect url...
Driver initialization 'default' FAILED! Retry 2 of 3 time - unable to achieve/connect url...
Driver initialization 'default' FAILED! Retry 3 of 3 time - unable to achieve/connect url...
HUGE STACKTRACE HERE

Verified with branch 7.3.17, fixed.


Simplify websocket code

All these functions:

connectToSocket
initialiseSocket
tryToConnect

seem overly complicated.
Can we delete all of them, start from scratch, and see how much we get for free with https://github.com/lukeed/sockette

Maybe sockette is not the right tool for the job, but I would like some library that can abstract away the reconnecting part.

I think we can just roll our own tiny library inspired by Sockette: https://github.com/lukeed/sockette/blob/master/src/index.js
If we copy any code, we should remember to include their MIT license.

I think a copy of Sockette, but allowing infinite reconnects and always reconnecting no matter the error is probably what we need.


locked_file has been moved to oauth2client.contrib

After https://github.com/google/oauth2client/commit/41eb75578f81876a53f1707cf1daf1eb89ddfc0b locked_file has been moved

I signed it!

subsumed by #182


[SPARK-10518][Docs]Update code examples in spark.ml user guide to use LIBSVM data source instead of MLUtils

I fixed to use LIBSVM data source in the example code in spark.ml instead of MLUtils

Can one of the admins verify this patch?

@y-shimizu Please also remove the MLUtils import if it is not used elsewhere.

@mengxr I fixed. And sorry, I was wrong to use val in Java.

@y-shimizu The changes to ml-ensembles look good. But we also need to update other spark.ml guides, e..g, ml-features.md and ml-linear-methods.md. Please find the list on JIRA.

@mengxr I updated ml-features.md and ml-linear-methods.md .

LGTM. Merged into master. @y-shimizu I don't know your JIRA account name. Could you ping me on the JIRA page so I can assign it to you? Thanks!


WIP: detect old attribute values

This addresses #73 .
This may not be correct, but limited testing shows it works for me.
Old attribute values are available to state trigger as binary_sensor.test.old.attribute_name.
I've also added change to func_args to indicate that a change was detected.
@state_trigger takes kwarg only_on_change (default: True)
I will continue to update this PR as I test.

This test code seems to do all the right things:
@time_trigger('startup')
def set_test_state():
    task.unique('set_test_state')
    while True:
        state.set('binary_sensor.test', 'on')
        task.sleep(15)
        state.set('binary_sensor.test', 'off')
        task.sleep(15)

@time_trigger('startup')
def set_test_attrib():
    task.unique('set_test_attrib')
    cnt = 0
    while True:
        cnt = cnt + 1
        if cnt > 10:
            cnt = 1
        state.set('binary_sensor.test', cnt=cnt)
        task.sleep(1)

@state_trigger('binary_sensor.test != "never_happens"')
def report_state_always(**data):
    log.info(data)

@state_trigger('binary_sensor.test.cnt != "never_happens"')
def report_attrib_always(**data):
    log.info(data)

@state_trigger('binary_sensor.test == "on" and binary_sensor.test.old == "off"')
def report_state_on_from_off():
    log.info('Is ON, was OFF')

@state_trigger('binary_sensor.test.cnt == 5 and binary_sensor.test.old.cnt == 4')
def report_cnt_5_from_4():
    log.info('CNT is 5, was 4')

@state_trigger('binary_sensor.test')
def report_state_by_eid(**data):
    log.info(data)

@state_trigger('binary_sensor.test.cnt')
def report_attrib_by_eid(**data):
    log.info(data)

@craigbarratt I don't know why these two tests are failing. I don't even know that it's related or how to tell what went wrong. Can you shed some light. (my knowledge of python tests is limited)

An issue of sorts.
This code...
@state_trigger('binary_sensor.test')
def report_state_by_eid(**data):
    log.info(data)

... before this PR meant to trigger on ANY change (state OR attribute) to binary_sensor.test. Now, with this PR, without adding only_on_change=False it only triggers if the actual STATE changes. This change will probably break some existing scripts.
What would be best is for a comparison style trigger (binary_sensor.test == "on") to have only_on_change=True but a non-comparison (just binary_sensor.test) to have only_on_change=False by default. But that gets weird.

Another option, which will create another possible collision in the binary_sensor.test namespace.
@state_trigger('binary_sensor.test') means trigger on any change to state OR any attribute. func_args will still just show binary_sensor.test and it's state for value and old_value.
@state_trigger('binary_sensor.test.some_attrib') means trigger on a change to that one attribute. func_args will show the attribute and the old and new values of the attribute.
@state_trigger('binary_sensor.test.state') means only trigger on a change to STATE. func_args will show binary_sensor.test as it does now.
So, we trigger on ANY change unless the user specifically indicates (via .state) that they only want to check the state.
Or... we add another kwarg of "state_only" or something to mean just the state.

For some reason I thought that the state_changed event didn't have the new and old state attributes, but indeed it does.  That's really good news.
I agree that @state_trigger shouldn't evaluate the expression if some unmentioned attribute changes.
I'd recommend we do the following.  The @state_trigger expression is only evaluated if variables that are mentioned in the expression (whether its value domain.entity,  attribute like domain.entity.attr, or virtual ones like domain.entity.old) have changed. That seems the most natural approach, and solves the current issue.  Since we keep track of which state variables or attributes are mention in the trigger expression, the notify code in state.py could do the comparisons to see what has changed.
I'd prefer the any-change trigger @state_trigger("domain.entity") to mean only any change to the value, not attributes.  While that would be a breaking change, I doubt many people are depending on that triggering on attribute changes right now. But I could be convinced otherwise - perhaps it could be a kwargs flag that defaults to the old behavior as you've done in #76. We could also support @state_trigger("domain.entity.attr") to mean any change to that attribute.
That's leaves a couple of open questions:

how to specify an any-change trigger.  It could be a kwarg option as just mentioned, some virtual attribute or a different decorator.  I haven't settled on the preferred method here - maybe the kwarg approach is best.
we now have an issue that a trigger could involve multiple variables changes if there are multiple attributes that change and the trigger expression mentions them.  What should we set the function arguments var_name, value and old_value to in that case?
do we support old attributes in the trigger expression, of the form domain.entity.old.attr?  I'd say yes.

One way to generalize and address the 2nd bullet (what to pass as value to the trigger function) is to make state variables evaluate to class instances that do the right thing, rather than fake them as being strings, but attributes still work.
This addresses a comment you made in #30:
    log.info(hasattr(input_boolean.test_1,'editable'))  # False
    log.info(input_boolean.test_1.editable)  #True

If we support that, it would be backwards compatible (the instance on its own would magically evaluate to a string), but would actually be an instance that provides attributes etc.
Then value and old_value could be the trigger state variables, and then inside the trigger function things like value.attr or old_value.attr would do the right thing, and would be based on the actual trigger state change, and not be subject to a race condition if the current attributes are fetched (which might have changed since the trigger), which is what happens now.
Then the @state_trigger function would be passed domain.entity and domain.entity.old set the real state objects, rather than just the strings currently, with attribute accesses falling back to fetching, which is prone to race conditions.

As this PR is now, it works as you describe, minus the "state variables are class instances" bit. A state trigger for binary_sensor.test will only trigger if the value changes. A state trigger for binary_sensor.test.cnt will only trigger if that attribute changes. And the func_args have var_name set to the state variables or the attribute variable based on the trigger, with value and old_value being set as expected.
Making "state variables class instances" gets into the tricky parts of pyscript code I don't fully understand yet (the Ast bits). So I'm not quite qualified to do that if that's the direction you want to take.

I'm also not yet sure how to make "state variables class instances".    If it's possible I think that's the way to go, although it's secondary to the bigger issue of only checking the state trigger when a referenced value changes.  It does make the value and old_value arguments work in a more elegant way, and would simplify the implementation.
The PR definitely got me thinking about how to do this.  Currently, however, it potentially calls State.update() multiple times - once if the value changes, and once for each attribute that has changed, independent of whether those are used in the @state_trigger.  It actually needs to call it just once, so that the trigger function is only evaluated once.  The logic about whether a referenced state variable or attribute has changed (ie, the ones mentioned in the @state_trigger) needs to move to the notify logic in state.py.
Let me think about the "state variables class instances" some more.

Ah ha. My test code was too narrow to see if the trigger function was being called more than once.
I'll see if I can make these changes. Thank you!

This PR essentially makes entity attributes act like entity states. Effectively, it issues its own version of a state_changed event for every state AND attribute. I think having a state change for every attribute is desirable. In fact, I wish Home Assistant did that already on its own. It would certainly elimate the tons of "feature requests" they've gotten for things that don't support attributes in the same way they support states.
Native Home Assistant Automation TEMPLATE Triggers don't have this issue because they approach the problem differently. If I write a Native Automation with two separate (NON TEMPLATE) triggers -- one for binary_sensor.test and one for binary_sensor.test.cnt -- it WILL trigger twice, by design.
However, if I instead use a single HASS Template Trigger in my automation, it would only fire once. This is because the HASS Automation keeps track of the True/False state of the Template Trigger, and only triggers if it changes from False to True. Future state_changed events that STILL evaluate to True do not cause a subsequent firing of the automation. With this in place, it doesn't matter if the changes come in one state_changed event or two, because the True/False evaluation of the Template will still only change once.
Personally, I PREFER the way pyscript works. I want it to trigger every time there is a change (not just when the False to True line is crossed). I can decide in my function if I want to do anything about that change. I can even keep track of False/True if I want, and only take action when it changes from False to True.
Something like a kwarg of "only_once=True" on state_trigger could invoke this False to True trigger logic if you wanted that to be in pyscript. Users can already do this themselves in code, but it wouldn't hurt to do it for them. I think it's common enough in general that it'd be nice to have, but not common enough for me personally that it's any trouble to just do it myself when I need it.
As it stands now, a state_trigger is, essentially, this:
@state_trigger('binary_sensor.test == "on" and int(sensor.cnt) > 5')

- trigger:
    platform: state
    entity_id:
      - binary_sensor.test
      - sensor.cnt
  condition:
    platform: template
    value_template: "{{ is_state('binary_sensor.test', 'on') and states('sensor.cnt')|int > 5 }}"

Though I haven't tested it, I imagine the above Home Assistant Automation would also trigger twice if the changes came close enough together.
For me, pyscript has the perfect blend of functionality and terseness of expression. The things I do most often are the most terse to write, the things I need less often are a bit more verbose but still possible. And having the trigger fire twice isn't a problem since I can account for it in my trigger function.
All of this to say, I think the trigger function running twice in race conditions, while not desirable, is okay. And it's happening now anyway without this PR so, while this PR doesn't fix that problem, it also doesn't make it much worse. If someone NEEDS to have just ONE trigger for any given entity this syntax is available:
@state_trigger('binary_sensor.test', only_on_change=False)

We can pass all of "old_state" and "new_state" from state_changed into the func_args. This would give users the ability to see what the old attribute value was in their code by doing:
@state_trigger('binary_sensor.test')
def a(**data):
  log.info(data['old_state'].attributes['cnt'])

"value" and "old_value" in func_args becomes redundant because the data is available in "old_state" and "new_state" as well.
But this would resolve the issue of not knowing what the old attribute value was inside the trigger function. This isn't the issue mentioned in #73, but is still a problem pyscript users have had trouble navigating.

I can make the call to State.update() only happen once per state_changed event, but State.update() breaks that into several queue.put()s anyway, so the end result would still be, potentially, multiple trigger function fires. And with a single call to State.update() we lose the ability to customize func_args for the values of the attribute.
If we go back to only notifying on state variables (domain.entity), we'd get back to only one trigger function fire per state_changed event. And users could still still test for binary_sensor.test.old.cnt in the state_trigger because we could pass those variables in to State.update(). And a check before the trigger function fired could make sure that SOMETHING changed (state or attribute) but we'd still see trigger function fires when an attribute changed when the user only cares about the state.
If we want to limit trigger function fires to only when the thing we care about actually changes value, then the most straight forward solution seems to be to notify on each attribute.

Perhaps we can call State.update() only once. We include all of old_state and new_state in the notification. Then trigger.py can compare them all, determine what changed, look at the variables in state_trigger and build func_args on its own based on what the user cares about.  However, if I have @state_trigger('int(sensor.test) > 5 and sensor.test.cnt > 5 and sensor.test.other > 5') and all three values change in the same state_changed event, which one would we trigger on?
Perhaps we always trigger with var_name = 'sensor.test', even if it was an attribute that changed?
We could include a dict like this in func_vars:
changes:
  sensor.test:
    old: 4
    new: 5
  sensor.test.cnt:
    old: 3
    new: 7
  sensor.test.other:
    old: 4
    new: 15

And we only trigger the function if they mentioned that specific thing in state_trigger. So a change to sensor.test.cnt with state_trigger('sensor.test') would not call the function.

replaced by #82


Fix broken jar

This PR intends to fix #94.

@henry-lp, now it's your turn to double-check. For now, forget about the integration with Repairnator. Let's first have a working jar, and then we see other issues later.
To sum up here, I found that the broken jar problem started when I upgraded Spoon, which was before starting using sonar-java to help in the detection of violations. So let's just use an older version of Spoon for the moment.
WDYT?

@fermadeiral can you also try downgrade spoon in #99 and check if it also works ? :)

@henry-lp, please, read again what I said in the other PR and in this one.

ok, then in that case, I have no problem merging this . WDYT ? :)

Thanks a lot for the fix and the constructive discussion on this tough problem.

@monperrus, thank you for merging.


Rajahentai

Source name
Rajahentai
Source link
https://rajahentai.xyz/
Language
Indonesian/Indonesia
Other details
NSFW✔
Please Add Soon
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] I have written a title with source name.
[X] I have checked that the extension does not already exist on the website extensions list or the app.
[X] I have checked that the extension does not already exist by searching the GitHub repository and verified it does not appear in the code base.
[X] I will fill out all of the requested information in this form.

please Add Soon

🙁

Can't add the extension?

🙁


Cache busting for the font URL

To allow defining a long cache header on the font files, it would be great if the reference in the CSS could provide some kind of cache busting, with a query string parameter updated when you update the font file (the query string parameter could use a hash of the file or a version number, depending of what is easier to you).

@jossef any plan to implement this ?

hey @stof
great suggestion.
any chance you submit a PR?
sorry for the late response

I'm not familiar with the way this package is built (I'm only consuming it, bundling the CSS in my project). So I don't know what would be the proper way to implement that cache busting (updating the cache busting param manually when the font is updated is a bad idea IMO, as it is too easy to forget)

I see that the build script already computes a hash of some files to detect changes. This might be used as the cache-busting param, if it is possible to inject it into the scss source (probably using a truncated hash only to be shorter).
However, currently, it seems like this hash accounts only for the list of icons, and so would not cache-bust things if a release is only changing existing icons.


Dynamic features with Navigation component

I know this might be a little outside of the scope for this sample, but I was wondering if there are any plans to integrate the navigation component in a way to showcase the recommended practices with dynamic features.
I've seen the other googlesamples repos and there is no demo available.
I've asked on SO but doesn't seem to be a lot of interest in the community, at least not for now :).
Thanks 🙏

P.S: Sorry if this shouldn't be an issue but I really didn't find any other place to request where it made sense.

They showed an example at google IO 2019  using the include tag, but it didn't work for me. when I try to include a navigation that exist in a dynamic feature module into the base module I get this error  "cannot resolve symbol".

What we talked about at Google I/O covered library modules. For dynamic-feature modules we're aiming to solve this as well. Here's a tracking bug for you to star.


[build-script] Make --infer to be True by default

This PR sets the --infer option to build-script to be True by default and adds a --no-infer tag to set it to False.
When the --infer tag is True it invokes build-script's dependency solver. Currently this is set to false by default.
I realized this when adding two new dependencies to the build system. I think it could make sense to always run the dependency solver despite the fact that it is not necessary given the current build configuration.

@CodaFi could you review this?
P.S. - This is my first PR to the Swift project! Please let me know if there is anything I should change about my PR and if it even makes sense for the Swift project :)

Pinging @compnerd and @gottesmm for review.

Pinging @compnerd and @gottesmm for review.

@justinfargnoli can you verify that this doesn't touch the preset path? There should be two paths in build-script one for non-preset path and one for preset path. If that is there, this LGTM.

Unfortunately, I don't think I'll be able to follow up on this PR, so I'm going to close it. Thank you @gottesmm for taking a look at it though.


Get built-in policy definition

I was used to use the getpolicydefinition function (https://www.pulumi.com/docs/reference/pkg/azure/policy/getpolicydefintion/) to access information about a Policy Definition, both custom and built in.
Now I would like to use Azure next-gen, but I don't see any way to get a built-in policy , the function getpolicydefinition https://www.pulumi.com/docs/reference/pkg/azure-nextgen/authorization/getpolicydefinition/ seems to be limited to a custom policy (I was able to use the function for a custom policy)
If I try with azure next-gen 0.6.0 :
import * as authorization from '@pulumi/azure-nextgen/authorization/latest'

// https://github.com/Azure/azure-policy/blob/master/built-in-policies/policyDefinitions/General/AllowedLocations_Deny.json
export const pdef = authorization.getPolicyDefinition({
    policyDefinitionName: 'e56962a6-4747-49cd-b67b-bf8b01975c4c',
})

I will get this error :
 Message="The policy definition 'e56962a6-4747-49cd-b67b-bf8b01975c4c' could not be found."

Is there a way to get a built-in policy with Azure next-gen ?
Note : bellow it's what I used with @pulumi/azure so it could be great to have the possibility to search by Name and a search by displayName with the azure next-gen function
import * as azure from '@pulumi/azure'

export const pdAllowedLocations = azure.policy.getPolicyDefintion({
    displayName: 'Allowed locations',
})

I suspect you would need this separate operation to get built-in policies: https://github.com/Azure/azure-rest-api-specs/blob/32b7b579549d417d5a799b4d0527f5ad9dd73fc3/specification/resources/resource-manager/Microsoft.Authorization/stable/2020-09-01/policyDefinitions.json#L182
At the moment, we don't expose stand-alone GET operations that aren't associated with a Pulumi resource (something creatable).
Your best bet, for now, would be using Azure SDK to retrieve this information, something similar to this example: https://github.com/pulumi/pulumi-azure-nextgen/issues/112#issuecomment-767600198


Ezoic integration results in validation error

Some users reported a Tag 'amp-story-bookend', if present, must be the last child of tag 'amp-story' validation error with Ezoic Analytics active on their sites.
The issue occurs as the Ezoic Analytics snippet is placed between <amp-story-bookend> and </amp-story>, with the amp-story-bookend element required to be placed right before the closing amp-story element (layout documentation).
One user contacted Ezoic, who are currently investigating, with a fix possibly incoming on their side.
This only started occurring recently, with impacted users below:

https://wordpress.org/support/topic/amp-bookend-improper-placement/#post-13938109
https://wordpress.org/support/topic/amp-validation-fail-tag-amp-story-bookend-must-be-last-child-of/
https://wordpress.org/support/topic/web-stories-no-longer-amp-validated/

Expected Behaviour

Steps to Reproduce

Screenshots

Additional Context

Web Stories Plugin version: 1.3.0
Ezoic Plugin Version: 1.3.13
WordPress Version: 5.6.0
Unable to reproduce on support side so far, using the Ezoic plugin or manually inserting the required Ezoic header code between  elements.

Labelling as support at present, with ticket opened from support on Ezoic website.

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria

Implementation Brief

For those impacted Ezoic provided an update to the ticket raised:
`I actually just got another update from our engineers right after I sent you my last message. The issue should now be fixed on our end with the plugin. They fixed the code to move all instances of amp-analytics before bookend tag.
`

For those impacted Ezoic provided an update to the ticket raised:
`I actually just got another update from our engineers right after I sent you my last message. The issue should now be fixed on our end with the plugin. They fixed the code to move all instances of amp-analytics before bookend tag.
`

Closing this as the issue was resolved.


Show warnings in query frontend

QFE currently does not parse warnings from downstream queriers. This commit fixes that by adding the field to proto messages and modifies the merge function to take warnings into account.

[x] I added CHANGELOG entry for this change.
[ ] Change is not relevant to the end user.

Changes

Show query warnings in query frontend.

Verification
Manual testing and unit tests.

I had to modify the cache middleware to also store warnings. This means cached responses will also return the warns produced when the original query was run.


Internal Tor needs to show if active and deactivate option

When using Tor within the app, we don't show if the in app tor is used we still just show the activate option. When active we need to show the option to deactivate (currently clicking activate again deactivates but doesn't show it on the screen)
Also when connecting the modal never closes even when we get connected. It should close once connected or on failure.

Verified this issue on keeper app v 1.3.1(433)


upgrade Jade dependency

Our Jade version is ancient and should be upgraded to latest.
But I know that it will break things.
I also know that I'm not interested in fixing it.  So... PR's accepted
(see old PR #1084)

Also jade is renaming to pug to avoid a lawsuit: https://github.com/pugjs/pug/issues/2184

https://github.com/mochajs/mocha/issues/2200#issuecomment-235493858


add theme Sakura

[ ] Read the theme publishing doc or plugin publishing doc.

@honjun
Thank you for your contribution :)


Is it possible to recover Bitcoin after it has been stolen or sent to the wrong address?

Hi there to all of you. If you have ever sent bitcoins to the incorrect wallet address or want to recover lost cryptocurrency from internet con artists, I wholeheartedly recommend<EMAIL_ADDRESS>Service. After contacting them, I was able to get my lost bitcoins back from an internet scammer in less than a day. They are the most skilled professional hackers I have ever encountered, and I am really appreciative of their help in getting back what I truly lost. When this occurs, you must move quickly to acquire professional help and retrieve the money that has been stolen. Even the safest bitcoin wallets might malfunction or have technical issues. If you experience this and lose access to your bitcoin, you can feel powerless. Fortunately, experts with the knowledge and skills of<EMAIL_ADDRESS>Contact them if you need help.

This is a cryptocurrency spam/scam! DO NOT REACH OUT TO THE ABOVE EMAIL ADDRESS, POSTED ABOVE!!!!. See #26657


Create viewed slider

Added "Viewed" slider to mark files as read in APIView.

You did not add all the files

No idea how I pulled that off. I did a new push and it looks like they're all there now, but let me know.


Typo in the Scaffold the movie model

There is a typo in the VSCode commands in the Scaffold section
dotnet-aspnet-codegenerator razorpage -m Movie -dc RazorPagesMovieContext -udl -outDir Pages\Movies --referenceScriptLibraries
Should be the following, without the extra '-' between dotnet and aspnet
dotnet aspnet-codegenerator razorpage -m Movie -dc RazorPagesMovieContext -udl -outDir Pages\Movies --referenceScriptLibraries
It may be the same issue in the macOS and Linux section, but the syntax may be different and I'm not aware.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6719f08e-3bd7-dc1a-71df-f2ef9fbca9d8
Version Independent ID: 7096fdb3-612e-9e00-bd0b-8ea4886a09ce
Content: Part 2, add a model
Content Source: aspnetcore/tutorials/razor-pages/model.md
Product: aspnet-core
Technology: aspnetcore-tutorials
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

They both work.

They both work.
>dotnet-aspnet-codegenerator --help
Usage: aspnet-codegenerator [arguments] [options]

Arguments:
  generator  Name of the generator. Check available generators below.

Options:
  -p|--project             Path to .csproj file in the project.
  -n|--nuget-package-dir
  -c|--configuration       Configuration for the project (Possible values: Debug/ Release)
  -tfm|--target-framework  Target Framework to use. (Short folder name of the tfm. eg. net46)
  -b|--build-base-path
  --no-build

To see more information, enable tracing by setting environment variable 'codegen_trace' = 1.
RunTime 00:00:00.07

Thank you, and I have to apologize as it does work fine. The command failed every time I included the hyphen between dotnet and aspnet - copied from the page or typed manually.
It now works, and I don't understand why so I can only assume it was something transient in my system.


Thoughts after reading LMDB docs again

I was going over the LMDB docs last night for inspiration. In LMDB, you have the remarkable property that separate processes (not just separate OS threads) can coordinate access to the database. On linux, it accomplishes this by keeping an interprocess posix mutex in the reader lock file. The readers all mmap the database file in read only mode (although they must all be able to write to the reader lock file). Since the readers and writer all share the same mmapped pages, you don't have to copy data to make it available. The operating system just pages in data as it's needed. Very cool. (You can also have the writer write directly to the mmapped memory, but this is dangerous.)
I was thinking about this mechanism and began to wonder if something like this would be appropriate for rotera. I eventually decided that I'm against attempting anything like this for the time being. Here are the concerns:

Rotera doesn't currently support multi-process access. All concurrency is currently handled by GHC's STM. To support reads from multiple processes (where the readers open the database in read-only mode), we would need to use posix mutexes like LMDB does. We would also need to be able to signal these readers when new messages showed up in the queue so that we could have something like the stream command that the stream sockets API supports. (I really dislike the idea of using timers for this kind of thing, e.g. having the reader check back every N seconds.)
How do we deal with an old reader? Currently, the only interface is stream sockets. If a reader is slowly reading some data that we want to overwrite because the queue has wrapped, we just memcpy that memory and keep sending the copy. And GHC's STM makes it really easy to do this. But, if instead a reader process were sharing mmapped file, we would be in a rather different position. We couldn't just copy it. The reader process would be using a pointer into the mmapped region. How does LMDB handle this? In LMDB, everything is basically a big append-only B+ tree. By looking at the reader lock file, the writer is able to tell what transaction the oldest reader is at. Any freed pages older than the oldest reader can be reclaimed. This works in LMDB because everything is segmented into pages (that's just how B+ trees work). It would not work for rotera because rotera uses a big rotating queue. To make this kind of thing possible in rotera, we would need to break the queue into segments, possibly 16MB each, and have the segments be ordered by an array somewhere else. While this would work, it's a totally different architecture than what we currently have. And users would only see any benefit it they put several readers on the same host as the writer.

In short, it would be rather complicated, and there is no immediate benefit. It's another project for another day. But if it were ever done, I believe that the stream-sockets interface to rotera might be able to be implemented as a reader process on top of this fancier architecture. In the case that multiple writers were needed, you'd use this process to coordinate the writers as well. In the case that only a single local writer was needed, we could take a page from LMDB's book and perform disk writes from the process doing the writing. (Well, if the stream-socket interface were used to coordinate writes, it would be the single local writer just doing this very thing.)

Today, I thought some more about other points in the design space.
There is a flag that fallocate supports called FALLOC_FL_COLLAPSE_RANGE. One thing it can be used for is to effectively truncate at the beginning of a file instead of the end. Through experimentation, I have found that the positions of existing file handle are maintained as one would hope them to be. This could be used to build an event log where each of the readers just maintains an open handle and calls read to pull and logs while advancing the handle's position. However, there are some problems. Slow readers are difficult to deal with. Also, it's hard to figure out where the end of the queue is. If you just say "open a handle at the end of the file", I'm not sure if you might get placed in the middle of an event. Also, every read creates a new copy in memory since you aren't using a memory-mapped file anymore. If you went with this strategy, you would probably want to interleave sizes with logs. Anyway, it sounds cool in theory, but the practical problems outweigh the novelty.
I do feel more convinced that the best design for this is having a single file that's been segmented. Probably not the 16MB segments I previously suggested, but maybe more like 256MB or 1GB. With 256MB segments, if you capped the event log at 4TB, you could get away with a fixed-size array that holds metadata for the 16384 possible segments. I'm not sure exactly what this metadata would include though, but even if were 256 bytes, that would put you at just 4MB of fixed overhead. Not terrible, but not great either. What about the other metadata, the metadata for offsets. It would have to live in the same segment as the data it described (and events absolutely cannot cross segments). One kind of annoying thing we currently have to deal with is the problem of the offsets wrapping around (either the offsets or the events may currently wrap around, which guarantees that we are wasting space). But I believe there's a more elegant solution here that becomes available when we are working with segments. What if the events starting at the beginning of a segment, growing upward, and the offsets started at the end, growing downward? When they meet (which implies that almost no space is wasted), the segment is considered full. A full segment may no longer be written to until it is evicted. Eviction happens when there are no readers and when space is needed for a new segment. Oddly, a slow reader (or a hung reader) can cause a newer segment to be evicted before an older one, but this does not actually cause any strange behavior. Any scheme that deals with slow readers will have to make it possible for a reader to observe a big skip, and this is no exception. The important thing is that no new readers are allowed to start reading from segment being held up by the slow reader.
Segments mean that growing and shrinking are possible, something that rotera currently does not allow. Hole punching with FALLOC_FL_PUNCH_HOLE makes a shrink possible while plain old fallocate makes it possible to grow. Allocating blocks brings the possibility of out-of-disk-space errors, so I'm not sure if it is actually a good idea to let this happen on demand.
Anyway, still all a separate project for a separate day, but I wanted to get these notes down somewhere.


ObstacleBasedValidStateSampler.h no such file or directory

Hey I am following the installation guide, where i exchanged the foxy parts to humble as I am using Ubuntu 22.04.
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release -DACADOS_WITH_QPOASES=ON -DACADO_CODE_IS_READY=ON -DWITH_IPOPT=true --packages-skip-regex archive --packages-skip vox_nav_control vox_nav_cupoch; \
The above command fails with
--- stderr: vox_nav_utilities
In file included from /home/andi/code/cpp/vox_nav/src/vox_nav/vox_nav_utilities/src/planner_helpers.cpp:17:
/home/andi/code/cpp/vox_nav/src/vox_nav/vox_nav_utilities/include/vox_nav_utilities/planner_helpers.hpp:20:10: fatal error: ompl/base/samplers/ObstacleBasedValidStateSampler.h: No such file or directory
   20 | #include <ompl/base/samplers/ObstacleBasedValidStateSampler.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

I had to set set(OMPL_INCLUDE_DIRS /path/to/vox_nav_ws/install/ompl/include/ompl-1.6) in src/vox_nav/vox_nav_utilities/CMakeLists.txt and src/vox_nav/vox_nav_planning/CMakeLists.txt

Thank you for providing your solution.
It seems that the colcon tool and ompl had a breaking change, causing this error.
Now, instead of source build, we use ompl-dev binary install; you may check the steps Github actions for a successful build.


parent:child relation not fully functional when using a durabledict other then memorydict

When using a pickled object the parent child relation is not fully functional. When you update the parent, the child doesn't pick up those changes, example:
from gutter.client.models import Manager
from gutter.client.models import Switch

from durabledict.memory import MemoryDict

class SpecialDict(MemoryDict):
	def _encode(self, value):
		return super(MemoryDict, self)._encode(value)

	def _decode(self, value):
		return super(MemoryDict, self)._decode(value)

gutter = Manager(storage=SpecialDict())

parent = Switch('parent')
gutter.register(parent)
child = Switch('parent:child')
gutter.register(child)
child.state = Switch.states.GLOBAL
child.save()
assert not gutter.active('parent:child')
parent.state = Switch.states.GLOBAL
parent.save()
assert gutter.active('parent:child')

I'm working on a clean fix at the moment.

this should be fixed in #23


Clarification on Max Output Tokens in LLaVA-1.6 Models

Question
I am currently working on a project that involves extracting information from technical diagrams. During my exploration of the codebase, I encountered references to parameters related to the maximum output tokens. I seek clarification on the specific roles of these parameters and their impact on the model's behavior.
Codebase References:

model_max_length in train.py (line no 86).
max_new_tokens in predict.py (line no 124).
model_max_length in finetune.sh (line no 44).

Questions:

What is the distinction between model_max_length and max_new_tokens?
Which of these parameters controls the maximum output tokens during inference?

Additional Query:
I am interested in increasing the default maximum output tokens (above 8k) to better suit the requirements of my project. Could you provide guidance on how to achieve this, if feasible?

model_max_length determines the max number of tokens a model can process and this includes system message, instruction and any response it generates.
max_new_tokens determines how many tokens it generates. The model will stop either it has generated an "End-of-Sequence" token or reaches that length.
Increasing the context length -- there are related research in NLP working on that. Also, you can switch the base LLM that supports longer context length as well :)

Understood, so max_new_tokens is equivalent to max_output_tokens. Is 2048 the max_new_tokens for the LLaVA 1.6 model?
Additionally, where can I locate documentation or instructions on switching to a different base Language Model (LLM)?
Thank you for your response, @haotian-liu.

Understood, so max_new_tokens is equivalent to max_output_tokens.
1)Could you provide information on the specific value of max_new_tokens for the liuhaotian/llava-v1.6-34b and liuhaotian/llava-v1.6-mistral-7b ?
2)I have tested mistral-7b-instruct-v0.2 on the Replicate website, and it successfully generated up to 8000 tokens during testing. Does this imply that the llava-v1.6-mistral-7b, can also produce up to 8000 tokens?
3)Moreover, when setting max_new_tokens to 8000, will this have any notable impact on the output or performance of the model?
Thanks for your reply @haotian-liu

Mistral instruct was trained to gracefully handle 32K tokens, while we haven't trained multimodal reasoning on that length -- that requires the model to "generalize" when generating any response above 4K.
Conceptually, as long as the total tokens are within 4K, it would be fine, so exist_tokens + max_new_tokens < 4K is the golden rule.

Hello, I am using liuhaotian/llava-v1.5-7b (4-bit version) and i am getting this warning:
Token indices sequence length is longer than the specified maximum sequence length for this model (4789 > 2048). Running this sequence through the model will result in indexing errors

I have computed some info from a perception model and I am trying to embed this information into text (I both tried to append this info to system message or directly to USER message). Anyway, both are really slow and
Do you have any suggestion on how to deal with this problem (maybe the best practice to embed this into the MLLM).
Thank you so much!!

model_max_length determines the max number of tokens a model can process and this includes system message, instruction and any response it generates.
max_new_tokens determines how many tokens it generates. The model will stop either it has generated an "End-of-Sequence" token or reaches that length.
Increasing the context length -- there are related research in NLP working on that. Also, you can switch the base LLM that supports longer context length as well :)

Does model_max_length include the image tokens?


config: remove strict mode from default SQL mode

What problem does this PR solve?
Relax the default SQL mode to remove the following strict mode components:

STRICT_TRANS_TABLES
NO_ZERO_IN_DATE
NO_ZERO_DATE
ERROR_FOR_DIVISION_BY_ZERO
NO_AUTO_CREATE_USER (said to be deprecated in MySQL docs)

The remaining enabled modes are:

ONLY_FULL_GROUP_BY
NO_ENGINE_SUBSTITUTION

What is changed and how it works?
Check List
Tests

Integration test

Side effects

Breaking backward compatibility

the default SQL mode is changed.

Related changes

Need to be included in the release note

Lightning no longer assumes strict SQL mode when importing data for better compatibility. If you need to enforce strict mode, please change the [tidb] sql-mode configuration.

Should we enable ALLOW_INVALID_DATES by default?


Clone feature states, feature value and feature segments on env clone

closes #172

Looks like we're missing quite a few tests now for each of the new clone methods. I thikn it might also be wise to put in a new full integration test similar to the ones that exist at the moment in tests/integration/.

I was planning to add  tests to the  test_models.py of the modules involved.  if we bring that part to test/integration we will have model tests at two places(unless we say that the test_models is deprecated) what do you think?

The integration tests aren't really testing the models, they should be testing the API interface only. So, for example, all the fixtures are created by hitting the relevant API endpoints, not by using the Django ORM.
There is definitely some duplication / overlap but I quite like testing the more complicated flows with integration tests so that we can verify that the actual client experience is correct.


Missing compiler warning when attempting to autodiff code that uses shared memory

Hi Slang team! I've been appreciating Slang's errors when I attempt to add [BackwardDifferentiable] to code that uses non-differentiable functions, but I think I ran into a case where a warning or error is missing.
Here's a function that broadcasts a value from thread 5 to all other threads in the group using shared memory (note: this code assumes there aren't reads of s_shared pending when thread 5 enters this function; otherwise it has a RaW hazard and needs an extra GroupMemoryBarrierWithGroupSync() at the top of the function):
groupshared float s_shared;

[BackwardDifferentiable]
float get_thread_5_value(float v, uint group_thread_id)
{
  if(5 == group_thread_id)
  {
    s_shared = v;
  }
  GroupMemoryBarrierWithGroupSync();
  return s_shared;
}

I'm calling it using a kernel that looks like this:
[shader("compute")]
[numthreads(128, 1, 1)]
void example_kernel(uint3 group_thread_id : SV_GroupThreadID, uint3 dispatch_thread_id : SV_DispatchThreadID)
{
  DifferentialPair<float> value = diffPair(g_in[dispatch_thread_id.x], 0.f);

  bwd_diff(get_thread_5_value)(value, group_thread_id.x, 1.0f);

  g_out[dispatch_thread_id.x] = value.d;
}

When compiled for CUDA (I haven't tested other targets), Slang 2023.4.1 produces code that always returns a differential of 0, without warnings or errors:
__device__ void s_bwd_prop_get_thread_5_value_0(DiffPair_float_0 * dpv_1, uint group_thread_id_1, float _s_dOut_0)
{
    DiffPair_float_0 _S1 = *dpv_1;
    if(5U == group_thread_id_1)
    {
        *(&s_shared_0) = _S1.primal_0;
    }
    __syncthreads();
    dpv_1->primal_0 = (*dpv_1).primal_0;
    dpv_1->differential_0 = 0.0f;
    return;
}

I think the correct backward derivative looks something like this; since this is complex and uses more shared memory than the original function, maybe it would be good for now to produce a warning or an error if someone tries to autodiff a function that accesses shared memory?
// For a variable x, let x_i be the value of x in the i'th thread in the group.
// Suppose we have y_i = v_5 and dR/dy_i = dResult_i.
// What's dR/dx_i?
// Well, it's sum(dR/dy_j dy_j/dx_i, j)
//         == sum(dResult_j (1 if i==5 else 0), j)
//         == sum(dResult_j, j) if i==5, else 0

// Used to compute the sum of dResult across all threads
// Side note: it would be nice if there was an intrinsic like WaveActiveSum for
// the whole group, but this isn't too much work to implement.
// Supposing the input has 128-thread groups and the GPU uses 32 threads/warp:
static const uint warps_per_group = 128/32;
groupshared float s_sum_shared[warps_per_group];

[BackwardDerivativeOf(get_thread_5_value)]
void get_thread_5_value_bwd(inout DifferentialPair<float> v, uint group_thread_id, float dResult)
{
  float warp_sum = WaveActiveSum(dResult);
  if((group_thread_id & 31) == 0)
  {
    s_sum_shared[group_thread_id/32] = warp_sum;
  }
  GroupMemoryBarrierWithGroupSync();
  if(5 == group_thread_id)
  {
    float group_sum = 0.0f;
    for(uint i = 0; i < warps_per_group; i++)
    {
      group_sum += s_sum_shared[i];
    }
    v = diffPair(v.p, group_sum);
  }
}

I've attached a few test .slang files and the results of transpiling them on Slang 2023.4.1 with -target cuda.
Thanks!
test-autodiff-shared-memory.zip

I agree that there should at least be a warning when a differentiable function is attempting to access a shared memory location.
We can support something like no_diff(g_shared = value); to allow the user to suppress the warning.


arc diff fails in git if submodules have changes

arc diff fails in git if submodules have changes
Arcanist seems to use git stash to try to bottle up working copy changes.  This doesn't work in the case of submodules, and will prevent users who have dirty submodules from using arcanist.

cd ../main
git submodule add ../sub/
git commit -m "Adding submodule."
echo "some change" >> sub/README
arc diff
At this point, arc diff will notice that the submodule has changes, and try to stash them.  Git stash doesn't affect submodules, so it arc will fail when it sees that the working copy is still dirty.
The "--ignore-submodules" switch to git status might be of use here.

Presume this identical to https://secure.phabricator.com/T3065


Update await_maybe

Replace iscoroutine with isawaitable in await_maybe towards the update to mypy 0.920.
Details
We are currently on version 0.910 of mypy, and are working towards updating to 0.920 in https://github.com/strawberry-graphql/strawberry/pull/1503.
In the new mypy version we get the following error in await_maybe:
strawberry/utils/await_maybe.py:12: error: Incompatible types in "await" (actual type "CoroutineType", expected type "Awaitable[Any]")

This PR modifies the function to use isawaitable instead of iscorotuine, which fixes the error.
Types of Changes

[ ] Core
[ ] Bugfix
[ ] New feature
[ ] Enhancement/optimization
[ ] Documentation

Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] I have tested the changes and verified that they work and don't break anything (as well as I can manage).

Thanks for adding the RELEASE.md file!

Here's a preview of the changelog:

This release modifies the internal utility function await_maybe towards updating mypy to 0.920.

Here's the preview release card for twitter:

Here's the tweet text:
🆕 Release (next) is out! Thanks to Yossi Rozantsev for the PR 👏

Get it here 👉 https://github.com/strawberry-graphql/strawberry/releases/tag/(next)

Thanks for contributing to Strawberry! 🎉 You've been invited to join
the Strawberry GraphQL organisation 😊
You can also request a free sticker by filling this form: https://forms.gle/dmnfQUPoY5gZbVT67
And don't forget to join our discord server: https://strawberry.rocks/discord 🔥


[tw] Localize Cluster for Traditional Chinese

Localize Cluster in Traditional Chinese
https://glossary.cncf.io/cluster/

Get Merged!


Exception: Subclasses need to define what the role start should be!

Hi i have some issue with my code. Can anyone help me please.

This is my model.

This is my code that i want to run using the model.

This is my error after i execute the model

Hey @Fahmie23, would you mind trying the same code with gpt-4? I think there's a small difference in the string that AzureOpenAI uses (vs. OpenAI) for the 3.5 series of models.

In the meanwhile, you can probably get around this by just using guidance.models.AzureOpenAIChat instead of guidance.models.AzureOpenAI.

In the meanwhile, you can probably get around this by just using guidance.models.AzureOpenAIChat instead of guidance.models.AzureOpenAI.

Thank you so much @Harsha-Nori. This solution work for my case. Thank you again


[ISSUE #8053] Return SYSTEM_BUSY if PutMessageStatus is OS_PAGE_CACHE_BUSY

Which Issue(s) This PR Fixes

Fixes #8053

It is correct to change the response code, but we need more discussion on whether client should retry on this scenario. Personally I think the client should retry by default to avoid producing failure.

I agree with you

It is correct to change the response code, but we need more discussion on whether client should retry on this scenario. Personally I think the client should retry by default to avoid producing failure.

There has been a lot of discussion about whether system_busy should be retried. I personally support retrying. For details, please refer to https://github.com/apache/rocketmq/pull/5845 @yuz10

@biningo merge develop to fix the ut

@biningo merge develop to fix the ut

okay, I have merged.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 42.88%. Comparing base (159a603) to head (e148e32).
Report is 5 commits behind head on develop.

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #8054      +/-   ##
=============================================
- Coverage      42.94%   42.88%   -0.07%
+ Complexity     10387    10369      -18
=============================================
  Files           1270     1270
  Lines          88694    88693       -1
  Branches       11401    11401
=============================================
- Hits           38092    38037      -55
- Misses         45914    45949      +35
- Partials        4688     4707      +19

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Hi @yuz10
#5845 already merged. Please review again.


vlc - impossible to install

Hello,
I try to install vlc but an error occur:
URL https://download.videolan.org/pub/vlc/3.0.8/win64/vlc-3.0.8-win64.7z is not valid
my computer: windows 10 (64bits), powershell

The issue is back again:
Whenever I start vlc, I get an additional empty console window like this:

And when I close it, VLC is also terminated :(
How can I fix this? :)

Any idea how to fix this? It's still happening with a scoop-installed VLC.

Any idea how to fix this? It's still happening with a scoop-installed VLC.

According to your screenshot, vlc's shim has been assigned to media files.
Cannot reproduct when actual vlc.exe assigned:

Reproduction:

install vlc
Assigning vlc's shim(<SCOOP>\shims\vlc.exe) to mp4 file
click the mp4 file
re-assigning the actual vlc.exe(<SCOOP>\apps\vlc\current\vlc.exe) via the context menu, but path of vlc.exe on resigitry seems not updated. (This may a Windows OS bug

Solution:

open registry editor, locate to Computer\HKEY_CLASSES_ROOT\Applications\vlc.exe, delete it. OR Remove-Item registry::HKEY_CLASSES_ROOT\Applications\vlc.exe
re-assigning the actual vlc.exe(<SCOOP>\apps\vlc\current\vlc.exe) for media files


WIP: Set object lock for volumes for cephfs encryption

Describe what this PR does
Provide some context for the reviewer
Is there anything that requires special attention
Do you have any questions?
Is the change backward compatible?
Are there concerns around backward compatibility?
Provide any external context for the change, if any.
For example:

Kubernetes links that explain why the change is required
CSI spec related changes/catch-up that necessitates this patch
golang related practices that necessitates this change

Related issues
Mention any github issues relevant to this PR. Adding below line
will help to auto close the issue once the PR is merged.
Fixes: #issue_number
Future concerns
List items that are not part of the PR and do not impact it's
functionality, but are work items that can be taken up subsequently.
Checklist:

[ ] Commit Message Formatting: Commit titles and messages follow
guidelines in the developer
guide.
[ ] Reviewed the developer guide on Submitting a Pull
Request
[ ] Pending release
notes
updated with breaking and/or notable changes for the next major release.
[ ] Documentation has been updated, if necessary.
[ ] Unit tests have been added, if necessary.
[ ] Integration tests have been added, if necessary.

Show available bot commands
These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

@Sunnatillo can you please check the CI failures?

@mergifyio rebase

Thank you for reviews @nixpanic, @Madhu-1.
I rebased to latest devel branch.

@nixpanic, I see fscrypt.Unlock call for RBD fscrypt. Do we need to do the same here as well?
do we need to
https://github.com/ceph/ceph-csi/blob/e71a95fece24de8de8f1de51201a7314af4f3480/internal/rbd/nodeserver.go#L467-L474

@nixpanic, I see fscrypt.Unlock call for RBD fscrypt. Do we need to do the same here as well?

No, for RBD that is not needed. RBD with a filesystem is never attached to more than one workernode at the same time, so initialization/unlocking of the filesystem will always be on a single node. It is different with ReadWriteMany RBD volumes in block-mode with encryption. The LUKS calls would benefit from a similar locking.

Is there a config to use the lock? We built an image from devel branch (https://github.com/ceph/ceph-csi/tree/devel) and tried it out. However pods failed to come up and we got the following error:
Warning  FailedMount             15s (x6 over 31s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-4642810a-7f00-4ae0-8583-ddbcee6ed314" : rpc error: code = Internal desc = Failed to lock volume ID 0001-0009-rook-ceph-00000000000000ad-d71d2949-db34-434e-8008-5497dad5c70b: rados: ret=-1, Operation not permitted

I have tested this and encountered the same error as @z2000l. @Sunnatillo, could you please document any additional settings required or anything we might be overlooking?

I can write up a ticket for the performance and for the permissions!