Image captioning target

I was trying to run image captioning model, but I get (latest error)

line 618, in get_data_statistics
    buck_idx, buck = get_target_bucket(buckets, len(target))
TypeError: object of type 'NoneType' has no len()

My target looks like this when I do head on it:
Closeup of bins of food that include broccoli and bread.
A giraffe eating food from the top of the tree.
A flower vase is sitting on a porch stand.
A zebra grazing on lush green grass in a field.
Woman in swim suit holding parasol on sunny day.
A couple of men riding horses on top of a green field.
They are brave for riding in the jungle on those elephants.
a black and silver clock tower at an intersection near a tree
A train coming to a stop on the tracks out side.
A couple of giraffe snuggling each other in a forest.

As a side note, it would be nice to provide some small example for image captioning that would actually work.

@lorisbaz

Can you please add more on the command that you are running? It is hard to extrapolate from that error.
Also, make sure you have the latest version of Sockeye (I do not see that line at 618 in the current version).

After extracting features in features/resnet-101, I tried the following training command (using your provided target):
python -m sockeye.image_captioning.train \
        --source-root features/resnet-101 --source source --target target \
        --validation-source-root features/resnet-101 --validation-source source --validation-target target \
        --batch-size 64 --initial-learning-rate 0.0003 --gradient-clipping-threshold 1.0 \
        --bucket-width 5 --max-seq-len 1:60 --fill-up replicate \
        --output models/ --encoder image-pretrain-cnn --rnn-num-hidden 512 \
        --rnn-decoder-state-init zero  --checkpoint-frequency 200 --weight-normalization

Maybe there is some hidden bug, can you please attach here the commands (including feature extraction and training) and full logs?

I run this command from Ipython:
DATA_DIR is directory with features (I've checked it and I think it's correct since feature extraction works)
!python -m sockeye.image_captioning.train \
        --source-root {DATA_DIR}/coco/train2014_features \
        --source train_features \
        --target captions_train2014 \
        --validation-source-root {DATA_DIR}/coco/val2014_features \
        --validation-source val_features \
        --validation-target captions_val2014 \
        --batch-size 64 \
        --initial-learning-rate 0.0003 \
        --gradient-clipping-threshold 1.0 \
        --bucket-width 5 \
        --max-seq-len 1:60 \
        --fill-up replicate \
        --output models/ \
        --encoder image-pretrain-cnn \
        --rnn-num-hidden 512 \
        --rnn-decoder-state-init zero \
        --checkpoint-frequency 200 \
        --weight-normalization

Also train and valid caption files are correct (I posted results of running head on one of them earlier).
I use sockeye version 1.18.56.

Do you also have the full log and error?

Full log:
[INFO:sockeye.train] The output folder /home/ubuntu/Projects/ml-sandbox/47lining-ml-sandbox/notebooks/image-search/models already exists, but no training state or parameter file was found. Will start training from scratch.
 New maximum length: (2, 61)
[INFO:sockeye.utils] Attempting to acquire 1 GPUs of 1 GPUs. The requested devices are: [-1]
[INFO:sockeye.utils] Acquired GPU master_lock.
[INFO:sockeye.utils] Acquired GPU 0.
[INFO:sockeye.utils] Releasing GPU master_lock.
[INFO:__main__] Training Device(s): gpu(0)
[INFO:sockeye.vocab] Building vocabulary from dataset(s): ['captions_train2014']
[INFO:sockeye.vocab] Vocabulary: types: 19212/19212/19212/19216 (initial/min_pruned/max_pruned/+special) [min_frequency=1, max_num_types=None, pad_to_multiple_of=None]
[INFO:sockeye.image_captioning.data_io] ===============================
[INFO:sockeye.image_captioning.data_io] Creating training data iterator
[INFO:sockeye.image_captioning.data_io] ===============================
[INFO:sockeye.utils] Releasing GPU 0.
[ERROR:__main__] Uncaught exception
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sockeye/image_captioning/train.py", line 392, in <module>
    main()
  File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sockeye/image_captioning/train.py", line 266, in main
    train(args)
  File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sockeye/image_captioning/train.py", line 315, in train
    output_folder=output_folder)
  File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sockeye/image_captioning/train.py", line 168, in create_data_iters_and_vocab
    preload_features=args.load_all_features_to_memory
  File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sockeye/image_captioning/data_io.py", line 229, in get_training_image_text_data_iters
    target_vocab=vocab_target)
  File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/sockeye/data_io.py", line 618, in get_data_statistics
    buck_idx, buck = get_target_bucket(buckets, len(target))
TypeError: object of type 'NoneType' has no len()

BTW I'm using EC2 with Deep Learning AMI. I run this command using mxnet_36 virtual env.

I couldn't reproduce this error, but I am trying to understand why you get that error.
It looks like target is None, however it shouldn't be empty since you were able to create the vocabulary with the same file captions_train2014 .
It seems that the SequenceReader is not able to parse correctly the file and the iterator target_sentences generates None sentences: https://github.com/awslabs/sockeye/blob/master/sockeye/data_io.py#L1091
Can you check if this function is working for you? https://github.com/awslabs/sockeye/blob/master/sockeye/data_io.py#L998
Maybe put a checkpoint on that line or print the output of list(get_tokens(line)).

@tdomhan Any hint?

@lorisbaz it seems like this function works OK
from sockeye import data_io
from itertools import islice

text_file_content = data_io.read_content('captions_train2014')

list(islice(text_file_content, 5))

Gives
[['Closeup',
  'of',
  'bins',
  'of',
  'food',
  'that',
  'include',
  'broccoli',
  'and',
  'bread.'],
 ['A',
  'giraffe',
  'eating',
  'food',
  'from',
  'the',
  'top',
  'of',
  'the',
  'tree.'],
 ['A', 'flower', 'vase', 'is', 'sitting', 'on', 'a', 'porch', 'stand.'],
 ['A',
  'zebra',
  'grazing',
  'on',
  'lush',
  'green',
  'grass',
  'in',
  'a',
  'field.'],
 ['Woman', 'in', 'swim', 'suit', 'holding', 'parasol', 'on', 'sunny', 'day.']]

Another reason why sequence is empty in https://github.com/awslabs/sockeye/blob/master/sockeye/data_io.py#L1091
can be because the word is not in the dictionary.
I just saw that you keep the punctuation "." and upper-case, e.g.,  "Woman". You should remove punctuation and lower-case every word. Maybe the problem is due to this.

You'd probably also want to check the output of sequence = tokens2ids(tokens, self.vocab) in https://github.com/awslabs/sockeye/blob/master/sockeye/data_io.py#L1087

Closing due to inactivity. Feel free to reopen for more questions.


LMS HEALTHPOINT FIX PLEASE <3

Can someone make a plugin to fix the health points in LMS when your osrs accounts hp level is 10. I die alot in lms because i just see green text and full red hp bar in my bottom right (when it should be RED and almost BLACK background because i am 14 hp!) but because my osrs accounts hp stats is 10, it shows it full hp and green text. can someone make a fix for this?

.

This is a bug with the vanilla client, not with RuneLite. We may try to add a fix for it in our client specifically, but recommend you report this as a bug to Jagex.

This was fixed in the June 22 update:

The Minimap's HP orb now shows your health in Last Man Standing as a proportion of 99, as that’s the base HP level used in the minigame.


Have auto and interval mining at the same time

Component
Anvil
Describe the feature you would like
Use case: Have fast transactions (= fast CI pipelines and tests) with automated software that reacts on events, like bots, indexers, etc
At the moment anvil allows either one mining mode or the other - auto, or interval. This is specifically signaled by the interval mining mode, that warns that auto is disabled.
Additional context
No response

this should be possible to support by adding a new mode that combines both of these:
https://github.com/foundry-rs/foundry/blob/d7eac74cfd786447cec9650048e2d2fac63fba0c/crates/anvil/src/eth/miner.rs#L102-L108

@mattsse is this high priority? I would like to work on it but it's my first time interacting with foundry so it will take some time to get this done

not high priority, the task here would be to add a new variant that is
Mixed {
  auto: ReadyTransactionMiner,
  fixed: FixedBlockTimeMiner
}

and then polls both

oh looks simple enough, thanks!

@mattsse hey, I created a draft pr with the new variant + it's pooling but I'm wondering if I should also add RPC handler for setting the mixed mining mode, when you have some time can you take a look? Thank you

Hi @caiquejjx thanks for picking this up, draft is looking good!
I think it makes sense to add RPC methods for this (anvil_setMixedmine(bool), anvil_getMixedmine(bool) matching anvil_setAutomine(bool) / anvil_getAutomine(bool)).
cc @mattsse

thanks @zerosnacks, I'll follow up with the RPC methods implementation then


Add picture icon to top page

On japanese top page, maybe we need "icon" pictures for "Github repository | Slack Community | Study Material (Japanese)" as in the previous webpage.
This makes it easy for readers to find those links, especially "quantum native dojo" which can serve as advanced use cases of qulacs.

Thank you this comments. I added the icons to the home page.


Add an example hook for GitLab integration

Thanks for the catch. I also noticed yesterday that the 404 case isn't detected correctly so that's fixed too now


Adding sub-cluster to original Louvain clustering

Hi there,
I tried to refine the Louvain clustering results and in particular I re-ran the clustering on a specific bigger cluster that was initially identified.
After this analysis, the original cluster was split in two and this makes sense considering the biological system we are studying. What is the easiest way to add this information to the original Louvain clustering results?
My naive approach of using something like:
adata.obs.louvain[adata.obs.index.isin(adata_sub.obs.index)] = '10'
Doesn't seem to be the correct one?
Thank you!

Hey! So you used restrict_to = ('louvain', ['1']) if you wanted to recluster '1'? You'll have a new clustering variable louvain_R in this case, which you can rename anything you like... checkout the documentation.
I guess that this is what you want.

It works perfectly, don't know hw I missed that in the documentation...thank you!


BeatLeader Sorting Plugins Don't Work on 1.37.0

BeatLeader 0.9.17 and Better Song List 0.40.
Under the "Sort" dropdown there used to be BeatLeader Plugins listed. I asked in the BL Discord and they said the issue may be with Better Song List.
If this isn't and issue with Better Song List, apologies for opening an issue.

Someone else at BL told me it's their issue now - closing.


Autocomplete stops solving table.property because of "if table.property then return end", within a function body

local tbl = {}
tbl.Property = nil :: string?

function test()
	if tbl.Property then
		return
	end

	tbl.Property.

end

The autocomplete breaks because there's return
Expected result would be

This doesn't happen for

It does know what type it is though.

This is working as intended.
Luau has observed the control flow in the if statement and determined that at the point that you're typing tbl.Property, it must be nil due to the original type annotation of string?. As such, there are no autocomplete suggestions being presented here.

This is working as intended.
Luau has observed the control flow in the if statement and determined that at the point that you're typing tbl.Property, it must be nil due to the original type annotation of string?. As such, there are no autocomplete suggestions being presented here.

hmm yeah
usually functions where you use return over assert, are a bit meh... because the autocomplete takes the first "return" argument of a function, so it returns nil if the first condition resolves into true, or it actually just skips through it


Adding addressautocomplete.js : This script file provide the autoc…

Address auto-complete functionally

Closing this PR, as it is holding an old branch open from being deleted.
If functionality is later added, recreate the PR and push the branch once again.


proposal: math/rand/v2: add ChaCha8.Read

Currently, if an application needs to generate a reproducible sequence of random bytes, it can choose amongst

math/rand.(*Rand).Read, which uses the predictable Go 1 random number generator; or
making a zeroReader that returns a sequence of zero bytes and using cipher.StreamReader to combine it with a *chacha20.Cipher, which is pretty inconvenient.

Since we now have a pretty good secure, deterministic, and specified RNG as math/rand/v2.ChaCha8, it makes sense to add a Read method to it.
func (*ChaCha8) Read(p []byte) (n int, err error)

Since it's on the *ChaCha8 value, it's not going to be accidentally misused in place of crypto/rand.
/cc @golang/security @golang/proposal-review

Compare #65562 which proposes adding Read to Rand instead, apparently on the assumption that it will have a ChaCha8 inside.

Another possibility:
// Bytes fills p with pseudo-random bytes from the default source.
func Bytes(p []byte)

// Bytes fills p with pseudo-random bytes.
func (*Rand) Bytes(p []byte)

Using the name Bytes avoids any possible confusion with crypto/rand.Read.
Also, it's simpler than implementing io.Reader; why return an error when there's no way the operation can fail?

Ah, yes, this is pretty much a #65562 duplicate, thanks @zephyrtronium. /cc @lukechampine
Even if it solves the name collision, I am not convinced by a package-level Bytes: without Seed, what's the point of having both that and crypto/rand.Read?
Eliding the error return is nice, but I think that having *ChaCha8 implement io.Reader is convenient. For example it can be swapped in for crypto/rand.Reader.

without Seed, what's the point of having both that and crypto/rand.Read?

Sometimes you want some random bytes for non-security purposes, and you don't want to juggle two rand packages or deal with a Read function that returns an error?

I think this proposal offers a performance advantage compared to #65562. The Read method on per-source RNGs can be optimized, whereas with #65562, Rand.Read would need to call each time the Source.Uint64 method, but as of ergonomics of use the #65562 seems better.

I'm also in favor of this proposal over my own. I stand by my position regarding the security of ChaCha8, but ultimately what matters is exposing a Read method somewhere, and directly tying it to ChaCha8 ensures that callers will know what they're getting wrt security.

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group
The proposal is to make the ChaCha8 type implement io.Reader.
It always returns the requested number of bytes and never returns an error.

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group
The proposal is to make the ChaCha8 type implement io.Reader.
It always returns the requested number of bytes and never returns an error.

Re-opening for proposal acceptance.

Uh, this is not the right issue.


Better Tests

Included the latest phpunit via composer.
PHPUnit: Convert notices, warnings, and errors into exceptions.
Moved Tests/ to tests/ to be aligned with the standard PHP project structure.

See https://github.com/phpexpertsinc/skeleton for how almost all standardized PHP projects are now organized (and also for more info about the unofficial standard).

Thanks for the PR , I am recently thinking about writing test on PEST like o have did for other project , and also improve coverage . Here is an example https://github.com/scrawler-labs/arca-orm/tree/main/tests


Finance library upgrade

All compatible css libraries are updated whose references was given in the jsp files mentioned in the excel sheet and the errors are resolved which caused due to updation of libraries

Hi vinoth sir,
The row class is added because with the update of bootstrap the form group
class is putting the different input boxes in columns, so to put them in
rows rather than in columns row class is used
Thanks & Regards
Khalid rashid
On Tue, Mar 12, 2024, 3:09 PM vinothrallapalli-eGov <
@.***> wrote:

@.**** commented on this pull request.
In
finance/egov/egov-collectionweb/src/main/webapp/WEB-INF/jsp/dishonor-cheque/dishonor_cheque_form.jsp
https://github.com/upyog/UPYOG/pull/982#discussion_r1521143602:

@@ -97,7 +97,7 @@ var chequeDDDateMendatoryMessage = "<spring:message code='msg.please.select.cheq

Why class row is added, normally we use only in view pages not on form
pages?
—
Reply to this email directly, view it on GitHub
https://github.com/upyog/UPYOG/pull/982#pullrequestreview-1930546499,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARRNSVCBHGDZLKK2LA3SJXTYX3EMHAVCNFSM6AAAAABEQIWBFGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSMZQGU2DMNBZHE
.
You are receiving this because you authored the thread.Message ID:
@.***>


Bug: Pot and Grass Shuffle in MM on Everdrive Issues

You know what these issues are but basically if you collect some (or all?) items in certain areas in MM that have pots and grass shuffled, then come back later, and collect a non-shuffled item (or any of the items that you already grabbed from pots/grass) and then transform, the screen will stay white for a certain amount of time. Sometimes this eventually goes through but other times it can lead to a hard crash. See Jeville's posts here: https://discord.com/channels/1004394204992118935/1153765170691190825
Your most recent attempted fix did not work.

Finally fixed


Avoid crash in compactOOOHead if wbl is nil

Trying to avoid this:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10dc1bd]

goroutine 82338 [running]:
github.com/prometheus/prometheus/tsdb/wlog.(*WL).NextSegmentSync(0x0)
        /backend-enterprise/vendor/github.com/prometheus/prometheus/tsdb/wlog/wlog.go:478 +0x3d
github.com/prometheus/prometheus/tsdb.NewOOOCompactionHead(0xc014a1d400)
        /backend-enterprise/vendor/github.com/prometheus/prometheus/tsdb/ooo_head_read.go:285 +0x3a
github.com/prometheus/prometheus/tsdb.(*DB).compactOOOHead(0xc001125d10)
        /backend-enterprise/vendor/github.com/prometheus/prometheus/tsdb/db.go:1165 +0x4f
github.com/prometheus/prometheus/tsdb.(*DB).CompactOOOHead(0xc001125d10?)
        /backend-enterprise/vendor/github.com/prometheus/prometheus/tsdb/db.go:1158 +0x96
github.com/grafana/mimir/pkg/ingester.(*userTSDB).compactHead(0xc00f371440, 0x6ddd00)
        /backend-enterprise/vendor/github.com/grafana/mimir/pkg/ingester/user_tsdb.go:201 +0x378
github.com/grafana/mimir/pkg/ingester.(*Ingester).compactBlocks.func1({0xc0064bc120?, 0xc008fdef28?}, {0xc001ce10d3, 0x6})
        /backend-enterprise/vendor/github.com/grafana/mimir/pkg/ingester/ingester.go:2111 +0x3c5
github.com/grafana/dskit/concurrency.ForEachUser.func1()
        /backend-enterprise/vendor/github.com/grafana/dskit/concurrency/runner.go:45 +0x14d
created by github.com/grafana/dskit/concurrency.ForEachUser
        /backend-enterprise/vendor/github.com/grafana/dskit/concurrency/runner.go:36 +0x125

This PR is ready to be merged https://github.com/prometheus/prometheus/pull/11962 and I think it may solve the issue as well.

Fix was merged upstream. PR to get it into mimir-prometheus https://github.com/grafana/mimir-prometheus/pull/453


ValueError: Target module QuantLinear() is not supported.

System Info
Hi,I was able to finetune a GPTQ model till 2 days back, but suddenly I have been getting the following error when executing the line: model = get_peft_model(model, config)
ERROR:
ValueError: Target module QuantLinear() is not supported. Currently, only the following modules are supported: torch.nn.Linear, torch.nn.Embedding, torch.nn.Conv2d, transformers.pytorch_utils.Conv1D.
platform: aws sagemaker
bitsandbytes: 0.41.0
torch:  2.1.0+cu121
peft:  0.6.0
transformers:  4.35.0
accelerate:  0.24.1
auto_gptq:  0.5.0
python: p310
Who can help?
No response
Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder
[ ] My own task or dataset (give details below)

Reproduction
from peft import prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig,LlamaForCausalLM,LlamaTokenizer
model_id = "TheBloke/CodeLlama-7B-Instruct-GPTQ"

tokenizer = AutoTokenizer.from_pretrained(model_id,padding_side="left", add_eos_token= True,return_tensors="pt",use_auth_token ="...")
#tokenizer.pad_token = tokenizer.eos_token  # Llama has no pad token by default
tokenizer.pad_token = '[PAD]'

quantization_config_loading = GPTQConfig(bits=4, use_exllama=True)
model = AutoModelForCausalLM.from_pretrained(model_id,
                                             #load_in_4bit=True,
                                             quantization_config=quantization_config_loading,
                                             device_map="auto")

model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model)

from peft import LoraConfig, get_peft_model
config = LoraConfig(
    r=4,
    lora_alpha=32,
    target_modules=["k_proj","o_proj","q_proj","v_proj"],
    #modules_to_save=["embed_tokens", "lm_head"],
    lora_dropout=0.2, #0.05 worked fine
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, config)
model.print_trainable_parameters()

Expected behavior
I would expect the model to load.

Thanks for reporting. I could not reproduce the error, your code passed for me. What versions of PEFT, auto-gptq etc. are you using? If not the latest, could you try updating?

@manideep-paa I just saw that your original code contained a HF token. I have edited your post and deleted it, but note that this token is now compromised. If it is your personal token or from your company, I would recommend to invalidate it and create a new one.


[BUG] move button need to be double click in Zena's device

IOS version 17.6.1

This bug fixed in ios dev version 2.2.8(2)


Upsert updates are not supported

It is related to #18 and #19. After initialization (npm start) on Let's Chat is successful, accessing to http://localhost:5000/login produces the following errors:
684219 [nioEventLoopGroup-3-2] DEBUG c.e.m.m.RequestMessageObjectHandler - Received message type: OP_QUERY, data: QueryMessage{clientAddress=<IP_ADDRESS>, clientPort=54056, requestId=7, database='letschat', collection='$cmd', numberToSkip=0, numberToReturn=-1, document={ "update" : "sessions" , "updates" : [ { "q" : { "_id" : "rY0RvMeYlwWHOy9o23zT_i8FfIkk1xZ5"} , "u" : { "_id" : "rY0RvMeYlwWHOy9o23zT_i8FfIkk1xZ5" , "session" : "{\"cookie\":{\"originalMaxAge\":null,\"expires\":null,\"secure\":false,\"httpOnly\":true,\"path\":\"/\"},\"passport\":{}}" , "expires" : { "$date" :<PHONE_NUMBER>335}} , "upsert" : true}] , "ordered" : true , "writeConcern" : { }}, returnFieldsSelector=null}
684221 [nioEventLoopGroup-3-2] ERROR c.e.m.m.RequestMessageObjectHandler - Error while processing request
com.torodb.torod.core.exceptions.UserToroException: Upsert updates are not supported
	at com.toro.torod.connection.DefaultToroTransaction.update(DefaultToroTransaction.java:210) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.torodb.mongowp.mongoserver.api.toro.ToroQueryCommandProcessor.update(ToroQueryCommandProcessor.java:197) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.eightkdata.mongowp.mongoserver.api.QueryCommandProcessor$ProcessorCaller.update(QueryCommandProcessor.java:226) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.eightkdata.mongowp.mongoserver.api.commands.QueryAndWriteOperationsQueryCommand$4.doCall(QueryAndWriteOperationsQueryCommand.java:69) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	...
684324 [nioEventLoopGroup-3-6] DEBUG c.e.m.m.RequestMessageObjectHandler - Received message type: OP_QUERY, data: QueryMessage{clientAddress=<IP_ADDRESS>, clientPort=54052, requestId=8, database='letschat', collection='sessions', numberToSkip=0, numberToReturn=-1, document={ "_id" : "rY0RvMeYlwWHOy9o23zT_i8FfIkk1xZ5" , "$or" : [ { "expires" : { "$exists" : false}} , { "expires" : { "$gt" : { "$date" :<PHONE_NUMBER>437}}}]}, returnFieldsSelector=null}
684326 [nioEventLoopGroup-3-6] ERROR c.e.m.m.RequestMessageObjectHandler - Error while processing request
com.torodb.torod.core.exceptions.ToroImplementationException: DocValue 2015-06-27T10:22:04.437 is not translatable to basic values
	at com.torodb.torod.core.subdocument.values.ValueFactory.fromDocValue(ValueFactory.java:91) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.torodb.translator.BasicQueryTranslator.translateGtOperand(BasicQueryTranslator.java:497) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.torodb.translator.BasicQueryTranslator.translateSubQuery(BasicQueryTranslator.java:428) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.torodb.translator.BasicQueryTranslator.translateSubQueries(BasicQueryTranslator.java:393) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	...
684488 [nioEventLoopGroup-3-7] DEBUG c.e.m.m.RequestMessageObjectHandler - Received message type: OP_QUERY, data: QueryMessage{clientAddress=<IP_ADDRESS>, clientPort=54053, requestId=9, database='letschat', collection='sessions', numberToSkip=0, numberToReturn=-1, document={ "_id" : "rY0RvMeYlwWHOy9o23zT_i8FfIkk1xZ5" , "$or" : [ { "expires" : { "$exists" : false}} , { "expires" : { "$gt" : { "$date" :<PHONE_NUMBER>604}}}]}, returnFieldsSelector=null}
684489 [nioEventLoopGroup-3-7] ERROR c.e.m.m.RequestMessageObjectHandler - Error while processing request
com.torodb.torod.core.exceptions.ToroImplementationException: DocValue 2015-06-27T10:22:04.604 is not translatable to basic values
	at com.torodb.torod.core.subdocument.values.ValueFactory.fromDocValue(ValueFactory.java:91) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.torodb.translator.BasicQueryTranslator.translateGtOperand(BasicQueryTranslator.java:497) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.torodb.translator.BasicQueryTranslator.translateSubQuery(BasicQueryTranslator.java:428) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.torodb.translator.BasicQueryTranslator.translateSubQueries(BasicQueryTranslator.java:393) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	...
684508 [nioEventLoopGroup-3-8] DEBUG c.e.m.m.RequestMessageObjectHandler - Received message type: OP_QUERY, data: QueryMessage{clientAddress=<IP_ADDRESS>, clientPort=54054, requestId=10, database='letschat', collection='sessions', numberToSkip=0, numberToReturn=-1, document={ "_id" : "rY0RvMeYlwWHOy9o23zT_i8FfIkk1xZ5" , "$or" : [ { "expires" : { "$exists" : false}} , { "expires" : { "$gt" : { "$date" :<PHONE_NUMBER>626}}}]}, returnFieldsSelector=null}
684509 [nioEventLoopGroup-3-8] ERROR c.e.m.m.RequestMessageObjectHandler - Error while processing request
com.torodb.torod.core.exceptions.ToroImplementationException: DocValue 2015-06-27T10:22:04.626 is not translatable to basic values
	at com.torodb.torod.core.subdocument.values.ValueFactory.fromDocValue(ValueFactory.java:91) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.torodb.translator.BasicQueryTranslator.translateGtOperand(BasicQueryTranslator.java:497) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.torodb.translator.BasicQueryTranslator.translateSubQuery(BasicQueryTranslator.java:428) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	at com.torodb.translator.BasicQueryTranslator.translateSubQueries(BasicQueryTranslator.java:393) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
	...

Hi,
First of all, thanks for your feedback. Talking about the issue... we know
upsert updates are not supported on ToroDB 0.23. When CRUD actions were
implemented, we decided to not implement that feature because the upsert
language supported by MongoDB is quite complex. We can easily implement a
subset of that language (and I am sure that subset would be enough for most
users) but we are focused on replication right now. The good new is that we
need to support upsert to support replication, so we hope we will release a
new ToroDB version in the following weeks that will support upsert and at
least a subset of replication features.
Please stay in touch,
Gonzalo Oritz
2015-06-27 3:31 GMT+02:00 Yuta Okamoto<EMAIL_ADDRESS>
It is related to #18 https://github.com/torodb/torodb/issues/18 and #19
https://github.com/torodb/torodb/issues/19. After initialization (npm
start) on Let's Chat is successful, accessing to
http://localhost:5000/login produces the following errors:
684219 [nioEventLoopGroup-3-2] DEBUG c.e.m.m.RequestMessageObjectHandler - Received message type: OP_QUERY, data: QueryMessage{clientAddress=<IP_ADDRESS>, clientPort=54056, requestId=7, database='letschat', collection='$cmd', numberToSkip=0, numberToReturn=-1, document={ "update" : "sessions" , "updates" : [ { "q" : { "_id" : "rY0RvMeYlwWHOy9o23zT_i8FfIkk1xZ5"} , "u" : { "_id" : "rY0RvMeYlwWHOy9o23zT_i8FfIkk1xZ5" , "session" : "{"cookie":{"originalMaxAge":null,"expires":null,"secure":false,"httpOnly":true,"path":"/"},"passport":{}}" , "expires" : { "$date" :<PHONE_NUMBER>335}} , "upsert" : true}] , "ordered" : true , "writeConcern" : { }}, returnFieldsSelector=null}
684221 [nioEventLoopGroup-3-2] ERROR c.e.m.m.RequestMessageObjectHandler - Error while processing request
com.torodb.torod.core.exceptions.UserToroException: Upsert updates are not supported
at com.toro.torod.connection.DefaultToroTransaction.update(DefaultToroTransaction.java:210) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.torodb.mongowp.mongoserver.api.toro.ToroQueryCommandProcessor.update(ToroQueryCommandProcessor.java:197) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.eightkdata.mongowp.mongoserver.api.QueryCommandProcessor$ProcessorCaller.update(QueryCommandProcessor.java:226) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.eightkdata.mongowp.mongoserver.api.commands.QueryAndWriteOperationsQueryCommand$4.doCall(QueryAndWriteOperationsQueryCommand.java:69) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
...
684324 [nioEventLoopGroup-3-6] DEBUG c.e.m.m.RequestMessageObjectHandler - Received message type: OP_QUERY, data: QueryMessage{clientAddress=<IP_ADDRESS>, clientPort=54052, requestId=8, database='letschat', collection='sessions', numberToSkip=0, numberToReturn=-1, document={ "_id" : "rY0RvMeYlwWHOy9o23zT_i8FfIkk1xZ5" , "$or" : [ { "expires" : { "$exists" : false}} , { "expires" : { "$gt" : { "$date" :<PHONE_NUMBER>437}}}]}, returnFieldsSelector=null}
684326 [nioEventLoopGroup-3-6] ERROR c.e.m.m.RequestMessageObjectHandler - Error while processing request
com.torodb.torod.core.exceptions.ToroImplementationException: DocValue 2015-06-27T10:22:04.437 is not translatable to basic values
at com.torodb.torod.core.subdocument.values.ValueFactory.fromDocValue(ValueFactory.java:91) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.torodb.translator.BasicQueryTranslator.translateGtOperand(BasicQueryTranslator.java:497) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.torodb.translator.BasicQueryTranslator.translateSubQuery(BasicQueryTranslator.java:428) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.torodb.translator.BasicQueryTranslator.translateSubQueries(BasicQueryTranslator.java:393) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
...
684488 [nioEventLoopGroup-3-7] DEBUG c.e.m.m.RequestMessageObjectHandler - Received message type: OP_QUERY, data: QueryMessage{clientAddress=<IP_ADDRESS>, clientPort=54053, requestId=9, database='letschat', collection='sessions', numberToSkip=0, numberToReturn=-1, document={ "_id" : "rY0RvMeYlwWHOy9o23zT_i8FfIkk1xZ5" , "$or" : [ { "expires" : { "$exists" : false}} , { "expires" : { "$gt" : { "$date" :<PHONE_NUMBER>604}}}]}, returnFieldsSelector=null}
684489 [nioEventLoopGroup-3-7] ERROR c.e.m.m.RequestMessageObjectHandler - Error while processing request
com.torodb.torod.core.exceptions.ToroImplementationException: DocValue 2015-06-27T10:22:04.604 is not translatable to basic values
at com.torodb.torod.core.subdocument.values.ValueFactory.fromDocValue(ValueFactory.java:91) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.torodb.translator.BasicQueryTranslator.translateGtOperand(BasicQueryTranslator.java:497) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.torodb.translator.BasicQueryTranslator.translateSubQuery(BasicQueryTranslator.java:428) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.torodb.translator.BasicQueryTranslator.translateSubQueries(BasicQueryTranslator.java:393) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
...
684508 [nioEventLoopGroup-3-8] DEBUG c.e.m.m.RequestMessageObjectHandler - Received message type: OP_QUERY, data: QueryMessage{clientAddress=<IP_ADDRESS>, clientPort=54054, requestId=10, database='letschat', collection='sessions', numberToSkip=0, numberToReturn=-1, document={ "_id" : "rY0RvMeYlwWHOy9o23zT_i8FfIkk1xZ5" , "$or" : [ { "expires" : { "$exists" : false}} , { "expires" : { "$gt" : { "$date" :<PHONE_NUMBER>626}}}]}, returnFieldsSelector=null}
684509 [nioEventLoopGroup-3-8] ERROR c.e.m.m.RequestMessageObjectHandler - Error while processing request
com.torodb.torod.core.exceptions.ToroImplementationException: DocValue 2015-06-27T10:22:04.626 is not translatable to basic values
at com.torodb.torod.core.subdocument.values.ValueFactory.fromDocValue(ValueFactory.java:91) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.torodb.translator.BasicQueryTranslator.translateGtOperand(BasicQueryTranslator.java:497) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.torodb.translator.BasicQueryTranslator.translateSubQuery(BasicQueryTranslator.java:428) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
at com.torodb.translator.BasicQueryTranslator.translateSubQueries(BasicQueryTranslator.java:393) ~[torodb-0.23-SNAPSHOT-jar-with-dependencies.jar:na]
...
—
Reply to this email directly or view it on GitHub
https://github.com/torodb/torodb/issues/21.

That's a great news! I'm looking forward to it.

@gortiz has upsert support been enabled?

Hi @graingert,
We were able to replicate oplog without upserts, so it is not yet implemented. To be honest, the abstract operation (upsert) is not very difficult to implement, but MongoDB supports some very specific operations (like $) that are not very well documented and a little bit tricky. As we came from the SQL world, we thought the operation was not very used and decided to not implement it until we had time to find a total implementation that were as expressive as MongoDB is. In the SQL world the upsert operation is not so important, as you can (almost always) replace it with a select-for-update + insert/update in a trasaction. Of course this is not possible on MongoDB, as it does not provide transactions and we totally undertand upsert is, in fact, a very important operation on the MongoDB ecosystem.
Right now we are pretty busy (funny thing, we are again improving the replication layer) so I cannot estimate when the upsert operation will be ready. On the other hand, a partial implementation that fails only when not supported operations (like $) are received should be not-very-difficult to implement by a new contributor (the the method that should be implemented is this one). You can try to do it if you feel brave enough ;-)
PS: Right now the update is, by far, the CRUD operation that has the worst implementation (in terms of efficiency) on ToroDB, so don't blame us if you found better ways to do it. We know it has to be improved, it is just a matter of time until we will do it.

We have added basic upsert support with #84 and #86 that allow to replace or create an entire document.

Hi all, is this functionality still on the roadmap?
A bit of background: I am trying to port our app (which uses MongoDB) to RaspberryPi 3. MongoDB limits storage to 2 GB when on 32-bit system. While RPi 3 has a 64-bit CPU, there is no 64-bit kernel drivers available (and no ETA for them), so I started looking for other solutions and found ToroDB. I love the solution (it would allow us to migrate slowly to PostgreSQL, which would be awesome - we are not that thrilled about MongoDB anymore), but the way I see it there are two major obstacles to this:
- protocol must be 100% compatible (not necessarily with the latest version!), which includes upserts
- performance (with the first test inserts were quite slow comapred to MongoDB)

We can live with performance problems (for now), but upserts and possible other incompatibilities are a deal-breaker for us. Note that there is no need to be compatible with latest MongoDB, it's enough to cover just a specific version.
Best regards!

Hi andyka.
Thank you for your interest. Yes, there is going to be some support for upsert soon. It will be limited to certain use cases but it will cover most of the usual usage, so will probably work for you.
There is a new version with many improvements due out soon. Please keep watching :)
Running >2GB mongo-like databases is a very interesting use case for us.

I close this issue, as it does not apply to the new version.
@andyka your case seems very interesting... can you wrote about it on our mail group?


Raise an error when repeated modules are found in configyml

Codecov Report
Patch coverage: 100.00% and no project coverage change

Comparison is base (8fe5fa0) 100.00% compared to head (b05ef90) 100.00%.

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #48   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            6         6
  Lines          827       827
=========================================
  Hits           827       827

Impacted Files
Coverage Δ

input/input.go
100.00% <ø> (ø)

runner/jobs.go
100.00% <ø> (ø)

dataset/dataset.go
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


--help should use less

The standard behavior for explicit help is to use the less utility or equivalent paginator rather than simply dumping the help text to stdout.  The reason for this is that it allows you to page through the help in a consistent fashion, then exit the help and return the terminal: your command history isn't scrolled off.

Clean

closing this as it is a duplicate of #7587
cc @tjprescott @JasonRShaver @binderjoe @johanste

reopening this as #7587 will be closed

I will be closing this for now as CLI team does not have the bandwidth to implement this in the near feature and there isn't a high demand for this feature.


I can't connect Adobe Analytics to Power BI Desktop

Is this missing all troubleshooting steps for Adobe Analytics connecting?  Basically I'm getting a bug or something and no workaround or troubleshooting steps are suggested in this article.
When trying to connect in Power BI desktop, It just gives me a grey screen popup when trying to connect it.  I already have it connected as an app to the web client UI for PowerBI - but when I try to do it from Desktop, the only way I can connect it is with the PowerBI datasource (online) and it only shows me just a handful of dimensions/metrics, whereas I can't just setup a datasource like I can with Google Analytics- and see more dimensions and metrics to create custom reports or get a larger selection of dimensions.  It's a little frustrating, because my adobe ID is connected to my adobe analytics account and I've tried using my password and my google account that is connected to my adobe ID to sign in and basically during authentication after password is entered or google ID is used, a blank grey window pops up and just sits there and does nothing and won't let me fully connect...  Is there potentially a troubleshooting step missing from this documentation?  Hence why I mention all of this.  (I am an admin of our adobe account by the way, and know that everything is setup on my end with my credentials correctly).
Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 384b4a18-dc3a-6a05-379f-8821b4a06513
Version Independent ID: b48f64dd-0d26-6cbb-3d3c-a29335f6ce82
Content: Connect to Adobe Analytics with Power BI - Power BI
Content Source: powerbi-docs/service-connect-to-adobe-analytics.md
Service: powerbi
GitHub Login: @SarinaJoan
Microsoft Alias: sarinas

Thank you for your question. This issue has been assigned to the author for review. He may need to refer you to customer support. He'll be in touch.

Hi @thesmarthomeninja - sorry about the trouble you're having. Sounds like it's an issue that'll require help from our support folks, rather than documentation. I'll also follow up with them to see if the docs can be clearer, but you'll want to initiate a support request (through the channel that's best for you - check out support.powerbi.com for a starting point) so they can troubleshoot your individual situation.

Sounds good, thank you sir!
Sent from Mail for Windows 10
From: David Iseminger
Sent: Tuesday, July 10, 2018 12:17 PM
To: MicrosoftDocs/powerbi-docs
Cc: Kalen Daniel; Mention
Subject: Re: [MicrosoftDocs/powerbi-docs] I can't connect Adobe Analytics toPower BI Desktop (#62)
Hi @thesmarthomeninja - sorry about the trouble you're having. Sounds like it's an issue that'll require help from our support folks, rather than documentation. I'll also follow up with them to see if the docs can be clearer, but you'll want to initiate a support request (through the channel that's best for you - check out support.powerbi.com for a starting point) so they can troubleshoot your individual situation.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

HI @thesmarthomeninja , did you ever get a solution for this? The same thing happens to me when I attempt to use the connector.

No, I could never figure it out. Seems like a broken feature or bug of some
sort. Thanks for responding, I was curious if it was just me!
On Aug 7, 2018 10:42 AM, "lfjhickory"<EMAIL_ADDRESS>wrote:
HI @thesmarthomeninja https://github.com/thesmarthomeninja , did you ever
get a solution for this? The same thing happens to me when I attempt to use
the connector.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/powerbi-docs/issues/62#issuecomment-411141405,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATbCnj6ZOp7o_BcxdRD7VmzHOgM3tvxWks5uOdFrgaJpZM4VFGgn
.

We are having the exact same issue with the grey screen on Adobe's end and the sign in with our experience cloud credentials just spins.

It's been broken for well over 6 months... Don't know who to contact about
it. I'm in the same boat as you all... Sorry
On Nov 21, 2018 12:31 PM, "mattpkelly"<EMAIL_ADDRESS>wrote:

We are having the exact same issue with the grey screen on Adobe's end and
the sign in with our experience cloud credentials just spins.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/powerbi-docs/issues/62#issuecomment-440799873,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATbCnv_Vnn3GWWGfG5FHgN8TbBwnH4sKks5uxbgRgaJpZM4VFGgn
.

I've submitted a ticket to Adobe Client Care for this and escalated it through our Account Manager to address with the Product Team.  In the meantime, I'm using ReportBuilder to publish data to Power BI.  So far it's working perfectly.

I have the same issue and think that it's due to the migration to the Adobeid model.

Hi,
it works for me on a PC under Windows7 but you need to have IE11 or higher (update is required).
The point here is that on windows XP, 7 and 8, you can't change the default registry key to open links from Applications like PowerBI with préfixe URL:RES.
So, except if you find a way to hack the registry entries to open this url in Google Chrome for example it won't work !
see : https://support.mozilla.org/fr/questions/1175204

I have the same problem, so basically we can not use this.

This issue has been reported a couple times in the Power BI Community too:

https://community.powerbi.com/t5/Desktop/Adobe-and-Power-BI-Connector/m-p/594653#M282361
https://community.powerbi.com/t5/Desktop/Unable-to-connect-Adobe-Analytics-to-Power-BI-Desktop-blank/m-p/498941/highlight/true#M232742

The blank grey window is supposed to be the one-time authorization step. If it's not appearing in the Power BI Desktop workflow you can try performing the authorization directly from the browser.

In a separate browser navigate to this link. The link is defined with the client_id used by the Power BI Desktop application that you need to authorize with Adobe.
Login with your Adobe ID.
The Authorization page will appear. Authorize access and then close the browser window after it progresses to a blank screen at "https://oauth.powerbi.com".
Then go back to Power BI Desktop and you will be able to successfully log in with your newly authorized Adobe ID.

The blank grey window is supposed to be the one-time authorization step. If it's not appearing in the Power BI Desktop workflow you can try performing the authorization directly from the browser.

In a separate browser navigate to this link. The link is defined with the client_id used by the Power BI Desktop application that you need to authorize with Adobe.
Login with your Adobe ID.
The Authorization page will appear. Authorize access and then close the browser window after it progresses to a blank screen at "https://oauth.powerbi.com".
Then go back to Power BI Desktop and you will be able to successfully log in with your newly authorized Adobe ID.

KingAdobe, thank you for helping out on this.


Adding colorize

Colorize is a simple and lightweight terminal coloring utility to quickly make your terminal outputs beautiful.

Looks good! :)


Bad range on bogus operator

let (+) = 1
let test = 1 + 2

(Error) Expected a function; got an expression of type "int"

The error points to 1 and not to +

Ha, this was fun. Inline rules in the parser were to blame:
commit 93c97b29f1241bb51ceb5356e01935733064b808
Author: Guido Martínez<EMAIL_ADDRESS>Date:   Sun Mar 22 00:25:54 2020 -0300

    parser: fix issue #1955

    Fixes #1955

diff --git a/src/parser/parse.mly b/src/parser/parse.mly
index f7b1906a1a..1279db3503 100644
--- a/src/parser/parse.mly
+++ b/src/parser/parse.mly
@@ -1164,7 +1164,11 @@ range:
   | op=OPINFIX0d
   | op=OPINFIX1
   | op=OPINFIX2
-     { mk_ident (op, rhs parseState 1) }
+     { mk_ident (op, rhs parseState 2) }
+/* ATTENTION! We're getting the range from the second component of the
+ * parse rule since this is *INLINED* below in the tmEqWith(X) rule, and
+ * there this production appears in the second position. Were this not
+ * %inline, it should be 1. See issue #1955. */

 %inline dotOperator:
   | DOT_LPAREN e=term RPAREN { mk_ident (".()", rhs parseState 1), e, rhs2 parseState 1 3 }

This seems like a limitation of ocamlyacc though, since menhir's docs state that

Finally, we remark that Menhir’s %inline keyword (§5.3) does not affect the computation of positions. The same positions are computed, regardless of where %inline keywords are placed.

So not sure if I should merge this fix or not, since we plan to move to menhir eventually, right?

I'm confused ...we are using menhir

Maybe I'm confused myself but it seems we use menhir only to preprocess the grammar and then we call ocamlyacc:
parse.mly: ../parser/parse.mly
ifeq ($(HAS_VALID_MENHIR), 1)
	@# TODO : call menhir directly when positions are fixed instead of
	@# letting OCamlbuild go through ocamlyacc
	$(MENHIR) --only-preprocess-for-ocamlyacc $< > $@
else
	$(error Correct version of menhir not found (needs a version newer than $(MENHIR_MIN_VERSION)))
endif

FStar_Parser_Parse.ml: parse.mly
	@# We are opening the same module twice but we need these modules
	@# open for the definition of tokens
	echo "open Prims" > $@
	echo "open FStar_Errors" >> $@
	echo "open FStar_List" >> $@
	echo "open FStar_Util" >> $@
	echo "open FStar_Range" >> $@
	echo "open FStar_Options" >> $@
	echo "open FStar_Syntax_Syntax" >> $@
	echo "open FStar_Parser_Const" >> $@
	echo "open FStar_Syntax_Util" >> $@
	echo "open FStar_Parser_AST" >> $@
	echo "open FStar_Parser_Util" >> $@
	echo "open FStar_Const" >> $@
	echo "open FStar_Ident" >> $@
	echo "open FStar_String" >> $@
	@# TODO: create a proper OCamlbuild rule for this production so that
	@# OCamlbuild knows how to generate parse.mly first (possibly using
	@# menhir) and removes the production as needed.
	ocamlyacc parse.mly 2> yac-log ### <--- HERE
	cat yac-log
	@if [ "0$$(grep "shift/reduce" yac-log | sed 's/^\([0-9]\+\).*/\1/')" -gt 6 ]; then \
	  echo "shift-reduce conflicts have increased; please fix" && exit 255; \
	fi
	@if grep -q "reduce/reduce" yac-log ; then \
	  echo "A reduce-reduce conflict was introduced; please fix" && exit 255; \
	fi
	cat parse.ml >> $@
	rm parse.ml parse.mli


Reorganize scripts folder and add READMEs

Added instructions on how to run the mock server as well as how to use the recording feature of the client. Also cleaned up some inconsistencies in key usage

There are test failures.


Tabulate Convergence test errors

This happens both when run locally and with CI. The tabulate_convergence_orders_multirate.jl test fails
Algorithm convergence: Error During Test at /Users/treddy/.julia/packages/SafeTestsets/raUNr/src/SafeTestsets.jl:30
  Got exception outside of a @test
  LoadError: type SplitFunction has no field α
  Stacktrace:
    [1] setproperty!(x::Function, f::Symbol, v::Float64)
      @ Base ./Base.jl:39
    [2] update_inner!

The last commit to main passed the CI. I triggered the CI with a branch with no code changes from main here on 12/17, and the test fails

Jobs in CI are also failing: https://github.com/CliMA/ClimaTimeSteppers.jl/actions/runs/12401526411/job/34621087934?pr=334

This issue is caused by changes to SciMLBase. If I use version 2.67.0, the tests pass, That version is before this PR was merged. It seems like changes to the remake function are the cause of the missing fields.


Saving Error

Error Code
Error: Too Many Requests

Error: Too Many Requests
    at Request.<anonymous> (https://homebrewery.naturalcrit.com/homebrew/bundle.js:78380:21)
    at Emitter.emit (https://homebrewery.naturalcrit.com/homebrew/bundle.js:1954:20)
    at XMLHttpRequest.<anonymous> (https://homebrewery.naturalcrit.com/homebrew/bundle.js:78674:10)
{
  "status": 429,
  "method": "PUT",
  "url": "/api/update/JnAJDKspCHdf"
}

### Your description of what happened:

I was just changing a text and the error happened

It's being looked at, thanks for the report.  Will update you once it's fixed

This problem should be resolved now.  If it persists for you, please followup in this thread.
Closing this comment as a duplicate.


parsl hangs when interchange fails to start

Describe the bug
When the interchange fails to start, parsl will hang.
in desc development branch, this particular error comes about because of address="localhost" in my local config, which became invalid as of PR #2828  - see trace below - but it is related to any interchange startup failure.
2023-07-20 08:31:59.529 interchange:128 HTEX-Interchange(53526) MainThread __init__ [DEBUG]  Initializing Interchange process
2023-07-20 08:31:59.529 interchange:134 HTEX-Interchange(53526) MainThread __init__ [INFO]  Attempting connection to client at <IP_ADDRESS> on ports: 55785,55909,55898
2023-07-20 08:31:59.530 interchange:147 HTEX-Interchange(53526) MainThread __init__ [INFO]  Connected to client
2023-07-20 08:31:59.530 interchange:31 HTEX-Interchange(53526) MainThread wrapped [ERROR]  Exceptional ending for starter on thread MainThread
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl/process_loggers.py", line 27, in wrapped
    r = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 638, in starter
    ic = Interchange(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/src/parsl/parsl/executors/high_throughput/interchange.py", line 171, in __init__
    self.worker_task_port = self.task_outgoing.bind_to_random_port(f"tcp://{self.interchange_address}",
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/zmq/sugar/socket.py", line 498, in bind_to_random_port
    self.bind(f'{addr}:{port}')
  File "/home/benc/parsl/virtualenv-3.11/lib/python3.11/site-packages/zmq/sugar/socket.py", line 302, in bind
    super().bind(addr)
  File "zmq/backend/cython/socket.pyx", line 564, in zmq.backend.cython.socket.Socket.bind
  File "zmq/backend/cython/checkrc.pxd", line 28, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: No such device (addr='tcp://localhost:54518')

A clear and concise description of what the bug is.
To Reproduce
parsl master  c39700bc2283500e0327ed3d7f00cf0056f2dfed plus this patch, then run pytest.
--- a/parsl/executors/high_throughput/interchange.py
+++ b/parsl/executors/high_throughput/interchange.py
@@ -614,6 +614,7 @@ def starter(comm_q, *args, **kwargs):

     The executor is expected to call this function. The args, kwargs match that of the Interchange.__init__
     """
+    raise RuntimeError("Deliberate hang")
     setproctitle("parsl: HTEX interchange")
     # logger = multiprocessing.get_logger()

Expected behavior
If the interchange doesn't start properly (or if it dies any time later on during execution), this should be a halting terminal error for htex, not a hang.
Environment
my laptop

Hi, I got the same issue when using v2023.8.7. Reverting back to v1.2.0 solved my problem.


Move to ebitengine/purego

@TotallyGamerJet I plan to move this repository to ebitengine/purego as the game engine's name is now Ebitengine. Please let me know if you are ready or not.

I plan to move this tonight in Japan time (Jun 10)

Finally done. https://github.com/ebitengine/purego


next: Implement login ids key whitelist

connects #917
connects #915

@kaojohnny I was testing this PR together with sdk js and found an issue.

See the above screen shot of my local db _auth_provider_password table.
The first row is signed up with login ID branch and the second row is signed before using the login ID branch. Both are signed up with Signup With Username with "username"="admin". Seems the migration is broken.

Hi @Steven-Chan, the above result is expected. The migration only altering column type from jsonb to text, so your previous row will remain.  Since skygear is not released, so I think it could be OK. Or we can delete all previous rows in migration script?

@kaojohnny Oh I see.
Please add comment as mentioned above, and I will merge this.

@Steven-Chan Thanks, Comment added.


All Swizzels

This does seem a bit more excessive than I was thinking. Vec4 has like... 1024 permutations there?
Swizzles don't normally include 0, perhaps that can be dropped? That'd bring Vec4 back to 256, which is a slightly more reasonable number. Starting swizzles with O also introduces an intellisense issue I don't think I'd want in there either. Swizzle spam that starts with WXYZ groups all the way at the end, and doesn't really interfere with the other members, but O puts a bunch of that spam in the middle.

I could drop swizzles starting with zero and also remove the pointless ones like xyzw

Or could turn 0 into _

Or could remove all zeros

Have changed swizzles to be a separate static class so people need a using declare of StereoKit.Swizzels to use it

Was debating on adding swizzles to expand a float up

could also defer the using types to be a bit more specific as well an example is using StereoKit.Vec2Swizzels

Ok, so I've slept on this a bit, and I have a few more thoughts.

Adding the entire permutation of swizzles to C# feels like an attempt to make the language something that it's not. Kinda like writing macros in C to make the language syntax different. This is generally something I'm wary of, and I typically prefer idiomatic code.
Swizzles may end up getting used as a substitute for more descriptive code, like your earlier example, where v.ZYX could be used as shorthand to rotate 90 degrees clockwise around the Y axis. A more appropriate piece of code would be Vec3.RotateY90CW(v), which is much more clear about what it's doing.
Code is not zero cost, and a large number of permutations will increase binary size. Probably not that much in this case, but this is something I track and optimize for.
The properties that SK does have in the existing API are primarily about converting from one vector type to another, rather than swizzling values around. So while this may be a debatable or tangential point, I'm inclined to say SK doesn't actually have swizzles to begin with.

So with that all that, I don't think swizzles belong in StereoKit's core. They can still be handy to have around though, especially when trying to port complex shaders to the CPU, so I'd propose putting them in the Tools folder instead? That way people can opt into them if they really want them, but they don't clutter anything up.
The best format for this would be a single C# file with all the swizzles that people can copy/paste, or right-click -> save to their own project. I did also like _ instead of 0, and would point out there's a few misspelled occasions of Swizzels that would need fixed too.

Moved Swizzles to one folder in the tools folder


Improve docs and error message to make it clear that create package requires Git

See more details in #388.
There is some commented-out code in packages/create/src/step-template.ts that would provide a more useful error message in the case where Git is not installed or not correctly configured, since degit requires Git.  (Ideally we could use an alternative to degit that doesn't require Git to be installed already, something like gittar, but that would require more investigation.)
Additionally, we should document somewhere that the create package currently requires Git to be installed, at least until such time that we find an alternative that doesn't require Git.

@chrispcampbell Thanks, installed git and got SDE to install up to and not including emscripten. Got this error msg:
√ Would you like to install the Emscripten SDK? » Install under parent directory (C:\Users\ug2\emsdk)
Downloading Emscripten SDK to C:\Users\ug2\emsdk
Local development detected, performing git pull...
Activating Emscripten SDK 2.0.34...
✖ Failed to install Emscripten SDK: Command failed with exit code 9009: python3 emsdk.py install 2.0.34
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
But Python is installed on on my path
What am I doing wrong? Could it be that my Python installation does not have the "3" after python?

@ulrgol: Thanks for your patience with all of this.  While we strive to make SDEverywhere work equally well on macOS, Linux, and Windows, you're one of the first Windows users who have tried the SDE quick start instructions, and there are a couple of Windows-specific bumps in the road.
It's unfortunate that the Emscripten SDK installation process requires Python (and also it's not good that this dependency is not documented; I'll add that to my list of things to fix).
If you already have Python installed on your machine, then you can follow the step that is described in the error message above (that error comes from Windows itself):

disable this shortcut from Settings > Manage App Execution Aliases

For more information on how to disable that Python alias, see this SO thread:
https://stackoverflow.com/a/66409838
Once you disable the alias, then the Emscripten SDK install should work.  If it doesn't, let me know and I can help.  Alternatively, you can try a manual install of the Emscripten SDK as described here.  I don't like that the manual install includes so many steps, which is why I built a simpler process into SDEverywhere's create script, but you can try the manual way if all else fails.

@chrispcampbell Thanks for your help! I solved the problme with a manual install of emscripten. Got your sample SIR model to work as it should. Then I tried my 'real' model and it failed to load. It is a large subscripted model and I tracked the problem to this equation
white meat dmd pp = IF THEN ELSE(region = cn, white meat dmd CN, white meat dmd func pp[region])
where region are the (10) subscripts and cn is one of them.
Apparently, SDE thinks cn is a variable because I get this error msg
ERROR: no var found for refID : '_cn'
I (think I) need this equation because for some regions the equation, in this particular case for _white_meat_dmd_pp differes from the 'default' equation I use for all the remaining regions.
Is there a workaround for this type of equation in Vensim? or in SDE? If you want to I can send the model, which works in Vensim, but it is large (~2000 equations, half with subscripts and exogenous data).
Thanks and best, Ulrich

Actually the Vensim equation is
white meat dmd pp[region] = IF THEN ELSE(region = cn, white meat dmd CN, white meat dmd func pp[region])

Seems like this is a parsing error in the conditional test of the IF THEN ELSE statement.
I'm not sure this matters, but are the [region] subscripts defined as lowercase or uppercase?  Most of the time two country codes are "CN" uppercase like you have in the variable name.
Also, this equation is one of several for a variable where you're trying to do define a particular value for a subscript, right?  Are there other equations for this model variable?  I think you might be defining the variable in a way that hasn't been tested in SDE before. If so, the fix might be to redefine the equation in Vensim.

@chrispcampbell @travisfranck  The subscripts are defined lowercase. And yes, this variable is one of several where I am defining a particular value for a subscript. The other equations are:
white meat dmd CN=  white meat dmd CN a* LN( GDPpp USED[cn] * UNIT conv to make exp dmnl) + white meat dmd CN b
white meat dmd CN a=   19.1013
white meat dmd CN b=   2.61946
white meat dmd func pp[region]=  white meat dmd func pp L[region] / ( 1 + EXP ( - white meat dmd func pp k[
region] * ( GDPpp USED[region] * UNIT conv to make exp dmnl - white meat dmd func pp x0 [region]) ) )
white meat dmd func pp k[region]=    GET XLS CONSTANTS('e4a-exo.xlsx', 'para' , 'c148' )
white meat dmd func pp L[region]=    GET XLS CONSTANTS('e4a-exo.xlsx', 'para' , 'c149' )
white meat dmd func pp x0[region]=    GET XLS CONSTANTS('e4a-exo.xlsx', 'para' , 'c150' )
white meat dmd pp[region]=   IF THEN ELSE(region = cn, white meat dmd CN, white meat dmd func pp[region])
I am not sure how to define these variables not as subscripted ones but as separate non-subscripted ones, because I would always have to use the region identifier to use one or the other going forward in the causal chain ... will ponder
Thanks

@ulrgol You should move this to the Discussion Board when you reply, if we still need to discuss.
My suggestion is that you re-write the equation using multiple equations in Vensim's equation editor.  You'll have one equation for CN and another for the rest of the regions.  You can find more information on Vensim's site here:
http://www.vensim.com/documentation/adding_equations.html
I suggest this because I know SDE support this, and I think it will do the same as your IF THEN ELSE statement.

There have been a couple more new users (on Windows) that have run into this issue because they didn't already have Git installed on their machine.  It looks like the create-astro package (which I used for inspiration for @sdeverywhere/create) started using the giget package as a replacement for degit.  The former doesn't depend on Git being installed locally, and it is more likely to be maintained in the future than degit.
I've filed a separate issue #464 about replacing degit with giget, and will close this one once I've confirmed that giget will be a suitable replacement.

I've merged the changes described in my last comment, so as of v0.2.13, the @sdeverywhere/create package now works even if Git is not installed locally, see #464 for more details.  This should reduce friction for non-technical users who are new to SDEverywhere.  Closing as resolved.


why not support iOS?

This is a business framework specifically for Android. And as far as I know, a similar framework for iOS is SwiftUI.


check for hyphen in flag value

This resolves #43 .

If the -s flag is supplied, the value for the flag is checked to see if it begins with a hyphen (-).
If so, the it is validated against a slice of valid flags for the application itself to make sure that the following arg is not actually a flag.
If it is not a flag value for the application, it will be used as the delimiter.
If the value for -s does not start with a hyphen, then the normal behavior will take place.

Going to refactor and use the standard lib flag package.  Shutting this down!


issue with perl when installing roadies with conda

Hi
I'm trying to install roadies with conda and faced an error with perl.  I would appreciate your help how to solve this.
conda create -n road python=3.9
conda activate road
conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge
conda install bioconda::roadies

This is the summarized output log:
Channels:
 - conda-forge
 - bioconda
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
## Package Plan ##
  environment location: /usr/miniconda3/envs/road
  added / updated specs:
    - bioconda::roadies
The following packages will be downloaded:
    package

The following NEW packages will be INSTALLED:
  about-time         conda-forge/noarch::about-time-4.2.1-pyhd8ed1ab_0
  aioeasywebdav      conda-forge/linux-64::aioeasywebdav-2.4.0-py39hf3d152e_1001
  aiohappyeyeballs   conda-forge/noarch::aiohappyeyeballs-2.4.0-pyhd8ed1ab_0
  aiohttp            conda-forge/linux-64::aiohttp-3.10.5-py39hcd6043d_0
  aiosignal          conda-forge/noarch::aiosignal-1.3.1-pyhd8ed1ab_0
  alive-progress     conda-forge/noarch::alive-progress-3.1.5-pyhd8ed1ab_0
..
The following packages will be UPDATED:
  ca-certificates    pkgs/main::ca-certificates-2024.7.2-h~ --> conda-forge::ca-certificates-2024.8.30-hbcca054_0
  libgcc-ng          pkgs/main::libgcc-ng-11.2.0-h1234567_1 --> conda-forge::libgcc-ng-14.1.0-h69a702a_1
  libgomp              pkgs/main::libgomp-11.2.0-h1234567_1 --> conda-forge::libgomp-14.1.0-h77fa898_1
  libstdcxx-ng       pkgs/main::libstdcxx-ng-11.2.0-h12345~ --> conda-forge::libstdcxx-ng-14.1.0-h4852527_1
  openssl              pkgs/main::openssl-3.0.15-h5eee18b_0 --> conda-forge::openssl-3.3.2-hb9d3cd8_0
  zlib                    pkgs/main::zlib-1.2.13-h5eee18b_1 --> conda-forge::zlib-1.2.13-h4ab18f5_6
The following packages will be SUPERSEDED by a higher-priority channel:
  _libgcc_mutex           pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex          pkgs/main::_openmp_mutex-5.1-1_gnu --> conda-forge::_openmp_mutex-4.5-2_gnu

Proceed ([y]/n)? y

Downloading and Extracting Packages:
InvalidArchiveError("Error with archive /usr/miniconda3/pkgs/perl-5.32.1-7_hd590300_perl5.conda.  You probably need to delete and re-download or re-create this file.  Message was:\n\nfailed with error: [Errno 22] Invalid argument: '/usr/miniconda3/pkgs/perl-5.32.1-7_hd590300_perl5/man/man3/Parse::CPAN::Meta.3'")
InvalidArchiveError("Error with archive /usr/miniconda3/pkgs/perl-dbd-sqlite-1.72-pl5321h031d066_1.tar.bz2.  You probably need to delete and re-download or re-create this file.  Message was:\n\nfailed with error: [Errno 22] Invalid argument: '/usr/miniconda3/pkgs/perl-dbd-sqlite-1.72-pl5321h031d066_1/man/man3/DBD::SQLite::VirtualTable::FileContent.3'")
...
InvalidArchiveError("Error with archive /usr/miniconda3/pkgs/perl-digest-hmac-1.04-pl5321hdfd78af_0.tar.bz2.  You probably need to delete and re-download or re-create this file.  Message was:\n\nfailed with error: [Errno 22] Invalid argument: '/usr/miniconda3/pkgs/perl-digest-hmac-1.04-pl5321hdfd78af_0/man/man3/Digest::HMAC_MD5.3'")
InvalidArchiveError("Error with archive /usr/miniconda3/pkgs/perl-xml-regexp-0.04-pl5321hdfd78af_3.tar.bz2.  You probably need to delete and re-download or re-create this file.  Message was:\n\nfailed with error: [Errno 22] Invalid argument: '/usr/miniconda3/pkgs/perl-xml-regexp-0.04-pl5321hdfd78af_3/man/man3/XML::RegExp.3'")

I also did conda clean -a but it didn't help.
Thank you in advance.

Hi, it looks like there might be an issue with your Miniconda installation. I’ve just tested the process on a different machine, and it worked fine for me. Could you try removing the current installation and reinstalling Miniconda3 on your system by following these steps?
rm -rf ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod +x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh

export PATH="$HOME/miniconda3/bin:$PATH"
source ~/.bashrc

Also, could you please let me know which machine you are trying to run this on?

Thanks for the response and the suggestion. Unfortunately, removing the whole conda does not seem practical for my case as I have other environments for other projects. I'm using an institutional machine of Rocky Linux, x86_64, CPU Intel Xeon.
I've also tried to install roadies in another machine of Red Hat Enterprise Linux, x86_64, AMD EPYC and the installation has stuck at this for a while. I'm wondering whether it can take a few hours?
$ conda install roadies
Collecting package metadata (current_repodata.json): - WARNING conda.models.version:get_matcher(556): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.7.1.*, but conda is ignoring the .* and treating it as 1.7.1
done
Solving environment: unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Solving environment: unsuccessful attempt using repodata from current_repodata.json, retrying with next repodata source.
Collecting package metadata (repodata.json): | WARNING conda.models.version:get_matcher(556): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.6.0.*, but conda is ignoring the .* and treating it as 1.6.0
WARNING conda.models.version:get_matcher(556): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.9.0.*, but conda is ignoring the .* and treating it as 1.9.0
WARNING conda.models.version:get_matcher(556): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.8.0.*, but conda is ignoring the .* and treating it as 1.8.0
done
Solving environment: |

Thanks for your help

Update:
In the second machine (RedHat), after around 1.5 hours, it finished with this
...
 zlib               conda-forge/linux-64::zlib-1.3.1-h4ab18f5_1
  zstd               conda-forge/linux-64::zstd-1.5.6-ha6fb4c9_0
Proceed ([y]/n)? y
Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: - | |done
Executing transaction: \
For Linux 64, Open MPI is built with CUDA awareness but this support is disabled by default.
To enable it, please set the environment variable OMPI_MCA_opal_cuda_support=true before
launching your MPI processes. Equivalently, you can set the MCA parameter in the command line:
mpiexec --mca opal_cuda_support 1 ...

In addition, the UCX support is also built but disabled by default.
To enable it, first install UCX (conda install -c conda-forge ucx). Then, set the environment
variables OMPI_MCA_pml="ucx" OMPI_MCA_osc="ucx" before launching your MPI processes.
Equivalently, you can set the MCA parameters in the command line:
mpiexec --mca pml ucx --mca osc ucx ...
Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via UCX.
Please consult UCX's documentation for detail.

I can see some files but it seems the installation wasn't successful.
 ls /wrk/miniconda/envs/road/ROADIES/
ASTER-Linux         config      drawing_github.png  metadata_conda_debug.yaml  README.md        run_roadies.py
build_env_setup.sh  Dockerfile  LICENSE             mkdocs.yml                 reference_trees  test
conda_build.sh      docs        Linux.zip           __pycache__                roadies_env.sh   workflow

because  if I run run_roadies.py from the downloaded github repo
 $ python ROADIES-main/run_roadies.py --cores 1
rm: cannot remove 'output_files': No such file or directory
Unlocking working directory.
snakemake --cores 1 --config mode=accurate config_path=config/config.yaml num_threads=0 --use-conda --rerun-incomplete
Config file config/config.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
CreateCondaEnvironmentException:
The 'mamba' command is not available in the shell /usr/bin/bash that will be used by Snakemake. You have to ensure that it is in your PATH, e.g., first activating the conda base environment with `conda activate base`.The mamba package manager (https://github.com/mamba-org/mamba) is a fast and robust conda replacement. It is the recommended way of using Snakemake's conda integration. It can be installed with `conda install -n base -c conda-forge mamba`. If you still prefer to use conda, you can enforce that by setting `--conda-frontend conda`.
sh: ./workflow/scripts/get_run_noconverge.sh: Permission denied
sh: ASTER-Linux/bin/astral-pro2: No such file or directory
sh: ASTER-Linux/bin/astral-pro2: No such file or directory
Traceback (most recent call last):
  File "/wrk/ROADIES-main/workflow/scripts/noconverge.py", line 141, in <module>
    num_gt = converge_run(
  File "/wrk/ROADIES-main/workflow/scripts/noconverge.py", line 75, in converge_run
    gt = open(roadies_dir + "/genetrees/gene_tree_merged.nwk", "r")
FileNotFoundError: [Errno 2] No such file or directory: 'output_files/genetrees/gene_tree_merged.nwk'
Traceback (most recent call last):
  File "/wrk/ROADIES-main/run_roadies.py", line 17, in <module>
    subprocess.run(
  File "/wrk/miniconda/envs/road/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', 'workflow/scripts/noconverge.py', '--cores', '1', '--mode', 'accurate', '--config', 'config/config.yaml']' returned non-zero exit status 1.

It seems astral-pro2 is not installed. Conda should have installed it..
$ astral-pro2
-bash: astral-pro2: command not found

$ grep astral  ROADIES-main/workflow/scripts/noconverge.py
       "ASTER-Linux/bin/astral-pro2 -t {1} -u 3

Thanks for your patience in installing the package. Looking at the snippets you provided, I see that the package has been installed successfully. However, the issue is probably caused by mamba and conda environmental conflicts.
Unfortunately, I have not faced this exact issue, but there is a possible fix that might solve this error. Could you please make a one-line change in the existing code in the downloaded repo and try running the pipeline again? If the solution works, I'll modify the bioconda recipe to update the ROADIES package, which will fix this issue permanently.
In the downloaded ROADIES repository, open the file noconverge.py inside the workflow folder (ROADIES/workflow/noconverge.py). At line 31, add one extra argument, --conda-frontend conda to the cmd command as shown below:
    cmd = [
        "snakemake",
        "--cores",
        str(cores),
        "--config",
        "mode=" + str(mode),
        "config_path=" + str(config_path),
        "num_threads=" + str(num_threads),
        "--use-conda",
        "--rerun-incomplete",
        "--conda-frontend", "conda"
    ]

Once you make that change in the file, please try rerunning the ROADIES pipeline using the following command: python run_roadies.py --cores 16.
Please let me know how it goes or if you still face further issues. We have newly added the bioconda package, so your feedback and suggestions will certainly be helpful in making our tool error-free and easily accessible for future users.

Sure. Yeah, having both mamba and conda in machine usually causes problems.  Probably you meant workflow/scripts/noconverge.py, so I've updated the code. Now it progressed further but at the end similar error. Please find here the full log  roadies_log_16sept.txt.
rule sequence_select:
    input: test/test_data/droYak2.fa.gz
    output: output_files/samples/droYak2_temp.fa
    jobid: 17
    benchmark: output_files/benchmarks/droYak2.sample.txt
    reason: Missing output files: output_files/samples/droYak2_temp.fa
    wildcards: sample=droYak2
    threads: 0
    resources: tmpdir=/tmp

..
We are starting to sample test/test_data/droYak2.fa.gz
./workflow/scripts/sampling/build/sampling -i test/test_data/droYak2.fa.gz -o output_files/samples/droYak2_temp.fa -l 500 -s 216 -e 229 -t 0.9
/usr/bin/bash: line 3: ./workflow/scripts/sampling/build/sampling: No such file or directory

..
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-09-16T231826.092826.snakemake.log
rm: cannot remove '/users/s/.pasta': No such file or directory
sh: ASTER-Linux/bin/astral-pro2: No such file or directory
sh: ASTER-Linux/bin/astral-pro2: No such file or directory
Traceback (most recent call last):
  File "/wrk/ROADIES-main/workflow/scripts/noconverge.py", line 142, in <module>
    num_gt = converge_run(
  File "/wrk/ROADIES-main/workflow/scripts/noconverge.py", line 76, in converge_run
    gt = open(roadies_dir + "/genetrees/gene_tree_merged.nwk", "r")
FileNotFoundError: [Errno 2] No such file or directory: 'output_files/genetrees/gene_tree_merged.nwk'
Traceback (most recent call last):
  File "/wrk/ROADIES-main/run_roadies.py", line 17, in <module>
    subprocess.run(
  File "/wrk/miniconda/envs/road/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', 'workflow/scripts/noconverge.py', '--cores', '1', '--mode', 'accurate', '--config', 'config/config.yaml']' returned non-zero exit status 1.

I'm also wondering whether it installs all the packages for every run.

Yes. the conda env road was activated when I ran the command line. (I don't have snakemake in my base conda env.)
From the log that I've sent earlier roadies_log_16sept.txt, it seems like snakemake is trying to create conda env again (and it doesnt use the activated conda env).
$ python run_roadies.py  --cores 1
Unlocking working directory.
snakemake --cores 1 --config mode=accurate config_path=config/config.yaml num_threads=0 --use-conda --rerun-incomplete --conda-frontend conda
...
Building DAG of jobs...
...
Creating conda environment workflow/envs/msa.yaml...
Downloading and installing remote packages.
...

Thanks for providing the inputs. Unfortunately, I’m not able to recreate the error on my end, so I haven’t been able to identify a solution yet. It might be helpful if we could sync up via any means (Zoom/Google Meet) to take a more detailed look at the issues you are encountering.
In the meantime, you might want to try an alternative installation method—such as using installation script if you have root access to the server or using Docker. All the instructions are provided in the repository's wiki or README.
Let me know your thoughts or if you have any questions!


false positives on PCs in network with Windows Server 2019

Systems: Microsoft Windows Server 2019, ca. 86 Windows 10pro Clients and handful Windows 11 pro
Using Yara rules, we get the following false positives (with some variations on some clients:
CobaltStrike_Resources_Artifact64_v1_49_v2_x_v3_0_v3_3_thru_v3_14 /media/5AC48A90C48A6E57/pagefile.sys CobaltStrike_Resources_Reverse_Bin_v2_5_through_v4_x /media/5AC48A90C48A6E57/pagefile.sys CobaltStrike_Resources_Artifact32_v3_14_to_v4_x /media/5AC48A90C48A6E57/pagefile.sys CobaltStrike_Resources_Dnsstager_Bin_v1_47_through_v4_x /media/5AC48A90C48A6E57/pagefile.sys CobaltStrike_Resources__Template_Vbs_v3_3_to_v4_x /media/5AC48A90C48A6E57/pagefile.sys CobaltStrike_Resources_Artifact32svc_Exe_v3_1_v3_2_v3_14_and_v4_x /media/5AC48A90C48A6E57/pagefile.sys
For full detailes reports see https://www.heise.de/forum/heise-Security/Themen-Hilfe/Desinfect/CobaltStrike-auf-Exchange-Server-gefunden/thread-7244749/

There are no Cobalt Strike detection rules in this repository. You've probably got those rules from somewhere else.

Oh sorry. Somebody send me this link. I will inform him.


Find Linux machine for GitLab CI/CD Runner

This project is heavy-weight. We need our own CI/CD runner to build Android/iOS binaries.

Done


Bitbucket 5.3.0 not yet on Docker Hub

Repo + docs updated, but docker hub does not yet have the release.

Build system slaves broke down.


Support for shielded transactions

@limpbrains see you're still active on this project, and I was hoping to use it for a zcash fork that only uses z_ operations.
Wanted to know if you were working on adding such functionality. I've forked it and have started trying to update things on my end, but I was wondering if you might have time to talk with me to clear up some questions I have.
I've added a boolean to the currency model for 'shielded_transactions', and have been able to get refill_address_queue working:
@shared_task()
def refill_addresses_queue():
    for currency in Currency.objects.all():
        coin = AuthServiceProxy(currency.api_url)
        count = Address.objects.filter(currency=currency, active=True, wallet=None).count()

        if count < settings.CC_ADDRESS_QUEUE:
            for i in range(count, settings.CC_ADDRESS_QUEUE):
                try:
                    if currency.shielded_transactions:
                        Address.objects.create(address=coin.z_getnewaddress(settings.CC_ACCOUNT), currency=currency)
                    else:
                        Address.objects.create(address=coin.getnewaddress(settings.CC_ACCOUNT), currency=currency)
                except (socket_error, CannotSendRequest) :
                    pass

Wanted to pick your brain if you're game...

Hi!
So it's a good start. I didn't try to implement this.
The next big question - does zcashd have the same API to get history of regular shielded transactions
I suggest you to try tests. Recently I've created tests for each currency. Each tests runs cryptocurrency daemon in docker and uses testnet to get real transaction.
To run tests simple install docker and run docker-compose up in django-cc dir
Zcash tests runs pretty long, because of the shielded transactions.
If these tests will pass on your branch, then it is possible to get this feature into django-cc
Zcash tests https://github.com/limpbrains/django-cc/blob/master/cc/tests/tests_zcash_regtest.py
Cheers

Yea, I believe in TDD. :)
I wish I knew enough about design patterns to be able to add a wrapper to add this functionality without modifying your app at all, I think the idiosyncrasies of this fork is going to be so different...
I'm still kinda new and learning what all the API calls do, but each z_address maintains it's own balance among the private total.
You can't query private txs like you can do on a public blockchain. There's no explorer on this thing at all, so if the daemon doesn't catch the operation... I think.
I'm going over the tests_mock module now. Can you tell me if the following tests are supposed to fail? It seems that the assertions should fail, IDK if there's something I'm missing here.
    def test_active_address(self):
        active = Address.objects.create(address='1111111111111111111114oLvT2', wallet=self.wallet, currency=self.btc, active=True)
        Address.objects.create(address='1AGNa15ZQXAZUgFiqJ2i7Z2DPU2J6hW62i', wallet=self.wallet, currency=self.btc, active=False)
        self.assertEqual(self.wallet.get_address(), active)

    def test_unused_address(self):
        unused = Address.objects.create(address='1Eym7pyJcaambv8FG4ZoU8A4xsiL9us2zz', wallet=self.wallet, currency=self.btc, active=False)
        Address.objects.create(address='LRNYxwQsHpm2A1VhawrJQti3nUkPN7vtq3', currency=self.ltc, active=True)
        self.assertEqual(self.wallet.get_address(), unused)

No, they should not fail. Both are fine


Update README with gene collapsing and insert size commands.

Thanks, this pull request does two things. It updates the documentation to add the last step to collapse down to the gene level fusion calls. It also renames the two python scripts to be prepended with pizzly-- those scripts are pretty generically named so they might introduce some clashes if we install them as binaries in the conda package. Prepending them with pizzly will get around that. Thanks!

I fixed the bioconda recipe to install the two python scripts as pizzly_ to match this pull request.

I added the shebang to the top of the python scripts which will make them executable via the users default python without prepending with python.

I also fixed a bug in the fragment length script, where the cutoff was set as the default even if a cutoff was supplied.

ping!


Global suggestion: don't use lib/ as a top-level directory

If newcomers use these repos as starting points, they might think lib/ is a good idea, and then be bitten when ReScript assumes lib/ is open for its own internal usage.
I personally don't know where this convention even comes from; at least in C-like environments lib/ would only ever be an output directory, not something that appears in the repo, except maybe after build. That is also consistent with how ReScript uses it. Sources should not be under top-level lib/, a directory traditionally for output or intermediate files.
Another drawback of having multiple top-level "source" directories (as opposed to grouped under src/), as often follows from lib/ (e.g. then also bin/, lib_lwt/, etc.) is that you have to list all of them if adding a recursive watcher. With src/, one just watches src/ (and maybe test/).

I've only ever really seen top-level lib/ sources in Mirage-style projects, but it's arguably wrong, and only "gliding by" because Mirage projects are not typically using ReScript, or autotools, or other tools assuming more "standard" conventions.

To add to the above, all of my projects personally are "younger" and very consciously and deliberately use src/, as well as older projects I maintained, which used src/ and kept using it very intentionally. So...:

Lwt
Bisect_ppx
Lambda Soup and Markup.ml, which are Dream-adjacent.
Dream itself, most relevantly.
odoc, which you already mentioned.

I didn't do a search over all projects, so I'm just listing what I immediately know and is somehow relevant, hence projects I am familiar with.

There is a clear pain point to using lib/

There is a very clear pain point to Dream-related projects using lib/ — as I mentioned, this conflicts with ReScript (not sure about Melange at this point) expecting to be able to ues lib/ as an intermediate files directory. This is no fault of ReScript, but is in accordance with long-established convention that a subset of the OCaml community did not heed.

One benefit of using {bin,lib}/ is also to separate the definition of the CLI from the rest of the code.

This benefit is not specific to bin/lib, as you can have src/lib/ and src/bin/, and basically all substantial projects do something like that (e.g. odoc, Bisect_ppx) without having to commit to bin/ (or lib/) being at the top level of the directory tree.
Other disadvantages of lib/ and bin/ at top level, besides the lib/ conflict:

A larger list of directories to watch.
A more messy top-level structure, forcing readers of a complicated project with many optional sub-libraries to wade through all of them looking for the non-sources.
A taller repository file list, which pushes down the README.
If a project has multiple sublibraries, a commitment to top-level directories forces strange things like lib_lwt, which quickly multiply causing a mess. They should have been all under a subdirectory to begin with, like src/ for the core, src/lwt/ for hte Lwt stuff.
What if a project has multiple binaries, like Bisect_ppx? Do we now have bin_ppx and bin_report? It looks like that underscore should be a slash, again strongly suggesting that subdirectories are the way to go. I understand this issue is not specifically about subdirectories, but the argument is that (1) there shouldn't be a top-level lib/, (2) all the benefits of a top-level lib/ are better achieved using subdirectories of src/ (or other favorite name) as necessary. You can still have all the right separate directories under src/.

I would be wary of statements about "young" and "popular' projects as grounds for anything, as there are many "young" projects following all sorts of conventions, including questionable retrogressive formatting conventions based on the drawbacks of other languages (e.g. Haskell and Elm, not saying they has only drawbacks, OCaml has its own other drawbacks) and older code bases, "novel" directory structures that haven't actually been tested against environments (ReScript), non-accessible usage of colors to represent distinctions (GitHub in a recent discussion we had), etc. Other "young" projects are avoiding these mistakes (but may well have yet other mistakes that they are making — no claim of overall superiority, just going issue by issue).
So I don't think "younger" or "popular" matters so much — bad conventions get picked up and established in environments that don't challenge them, etc. Again, not suggesting that projects that get this issue "right" according to me, don't suffer from the same problem in other respects according to me or others.

So, again, I don't have an opinion on which one is better, there seem to be pros and cons to both conventions.

I'm only aware of one pro so far, which is a flatter directory structure is more immediately parsable in terms of "what is everything that is in this repo." However, it can almost equally be well hidden behind only one level of indirection (src/), or retained flat, just without a directory named lib/, which is the specific point that will tend to bite some people who do Web development.

We can collectively (the OCaml community) agree to use a different convention

The ReScript community has already agreed to use a convention where lib/ is not a source directory. As I said, I think that's fine, and the novelty of lib/ as a top-level source directory was always something a bit likely to run into trouble eventually.

I didn't do a search over all projects, so I'm just listing what I immediately know and is somehow relevant, hence projects I am familiar with.

I just recalled, Daniel Buenzli's libraries are quite popular, and they seem to follow the "standard" src/ convention and do it quite beautifully. That includes cmdliner itself:

cmdliner
logs
fmt
The "younger" brr, which is planned for a full-stack example in Dream.
uutf
etc.

And apart from lib/ being an example that is dangerous to users of Dream with ReScript, if Dream ever does https://github.com/aantron/dream/issues/72 "Port Dream to Node," and wants to also port over some projects from its ecosystem, almost the first thing that will have to be done is that those projects which use lib/ will have to be adjusted somehow, or hacks will need to be introduced.

src/ also is better for quickly figuring out blame that has to do with the main source (as opposed to tests, etc.) With multiple top-level source directories, you can't immediately do that.
In general, src/ tends to be tidier, since it groups all of the "meat" of a project apart from all the "other stuff" that a project must also have. Again, you can (and practically all projects do) still separate the libraries and the binaries, etc. dune conventions encourage that (and it was always the right thing to do, even back in the day). Project top-level structure with src/ doesn't grow and mutate over time as a project chooses to drop or support a novel environment (like Lwt, Async, JS, etc.) — the repo is always tidy, and that growth does not spawn a whole new subdirectory branch of blame to suddenly look at. You can look at hte git history of src/ and see all of it. Conversely, you can easily miss that there was a lib_XYZ directory for a while, if you just didn't know to expect that it was there in 2017-2019.

That's a lot of rather convincing arguments, thank you for taking your time to share 🙂
There are two issues at hand here I think:

The use of lib/ to name a top-level directory
The use of multiple top-level directories

I'm actually already using subdirectories of {bin,lib}/. Typically, I would do:
bin/
  - binary_1/
  - binary_2/
  - ...
lib/
  - library_1/
  - library_2/
  - ...

So the top-level does not grow. So I think the questions are:

Should lib/ be named src/ instead?
Should bin/ be a subdirectory of src/, or otherwise named differently.

FWIW Daniel Buenzli's convention seems to be to split the binary code to another top-level directory as well (e.g. Carcass).
I'm not sure if you're arguing for a single top-level directory containing all of the sources, or if only the prevention of an ever-growing number of top-level directories? Or in other words, does the src/ and src-bin/ convention in the Carcass repository seem like a good one to you?
I could get convinced that src/ is a better convention, but again the most important factor and objective to me is establishing a consistent convention in the ecosystem (I'm not sure if ReScript counts, they seem to inherit conventions more from the JS ecosystem than the OCaml ecosystem).
You've evidently put a lot of thought into this, so maybe there is enough material to open this conversation with the rest of the community?
Would you be willing to ask the question as a Discourse post?
I realized this is a big-time investment for such a question, but the conclusion drawn would potentially make me revisit all of the Spin templates I have created (in addition to the Dream libraries obviously), which will impact a potentially large number of beginners, especially now that Spin is getting more and more recommended.
So I think taking the time to try and gather the community around a convention is worth it.

The most important thing is not to use lib/. src-bin/ is strange (again, it's pointless IMO, could use src/bin, and if you have two binaries, src/foo and src/bar is just as flat). But it's not something I would even raise an issue about.
I don't have a goal of getting the whole ecosystem behind this, though I would separately be very happy to see the whole ecosystem move away from lib/ for new projects. The minimal and justifiable goal is that it is not used with Dream. Dream also doesn't have a goal of following "the community" or waiting for "community" decisions (who is that? how would that thread look? when has "the community" "decided" something?). Which isn't to say that Dream is in any way against the community — it's just not influenced by considerations of relatively distant projects, nor should they be influenced so directly by Dream.
I would like templaters used with Dream to adopt conventions sensible for the users of Dream, however. Such projects have more influence on "the community" at large than one-time discussions anyway, so if you'd like an effect of making the ecosystem more consistent in a global way, it could be indeed advanced through these projects.
If you'd like to poll the community before proceeding with all of Spin (I'm only asking about spin-dream, dream-cli, dream-livereload, etc), you're welcome to open a topic, send me a link, and I will comment with my point of view.
Dream can't ignore ReScript :)

Carcass is a good example, though. If you believe it is best to keep lib and bin at the top level, if you are willing to adopt Carcass convention and call them src/src-bin, then that will, of course, no longer conflict with ReScript.

Another example, in this case where top-level bin/ for sources is problematic.
As the traditional (and reasonable) name for the output directory for binaries (and as used in system directories), I use bin/ in the root of the Bisect_ppx npm package to store pre-built binaries for Linux and Mac. It would be unfortunate to have to come up with some odd name for this, if I had started with a template that used bin/.
There are several other projects in ReScript/Melange that distribute pre-built binaries.
In summary, bin/ is for staging binaries you are about to install, not for sources of binaries.
Likewise, lib/ is for staging libraries you are about to install, not for sources of libraries.
One source for the confusion that developed in some corners of OCaml might be that JavaScript and other non-compiled languages install sources as libraries, hence they have a tendency to put sources in lib/. This is inappropriate for OCaml.

Thanks a lot for the additional arguments @aantron!
I haven't personally been impacted by the problems you mentioned, but as I said, I don't have a strong opinion on one or the other. You seem to have quite a good case for not using lib/, so let's change the convention. I'll take some time to think of a new one and will update my different repositories (spin-dream included) once I've found one I'm happy with. I'm open to your directory structure suggestions in the meantime 🙂

BTW I guess it would make sense to move this issue to spin-dream.

I've recently updated the Spin templates to use src instead of lib. You can find the template here: https://github.com/ocaml-templates/spin-dream
I think we can close this issue then, but don't hesitate to open a new one on the Spin template repo if you have other suggestions/feedback!