Performance enhancements for getting timestamps and calculating elapsed times

We currently have our own get_timestamp function for getting the current timestamp and get_elapsed_time for calculating the difference between two timestamps as milliseconds.
>>> t1 = get_timestamp()
>>> t2 = get_timestamp()
>>> print(t1)
20220308 12:37:43.875
>>> print(t2)
20220308 12:37:49.022
>>> get_elapsed_time(t1, t2)
5147

We should replace these custom functions with functionality provided by Python's standard datetime module. In practice we could get the current time by using datetime.now() and could calculate the elapsed time by simply subtracting datetime objects:
>>> d1 = datetime.now()
>>> d2 = datetime.now()
>>> print(d1)
2022-03-08 12:43:56.109349
>>> print(d2)
2022-03-08 12:44:03.567573
>>> delta = d2 - d1
>>> round(delta.total_seconds() * 1000)
7458

One benefit of using the standard datetime module would be that we could remove lot of our custom code. A bigger benefit is performance measured here using timeit:
>>> timeit(lambda: get_timestamp())
1.0487582740024664
>>> timeit(lambda: datetime.now())
0.2848877740034368
>>> timeit(lambda: get_elapsed_time(t1, t2))
5.105318886999157
>>> timeit(lambda: round((d2-d1).total_seconds() * 1000))
0.24636966099205893

As can seen from the examples, the difference in performance is pretty big when getting the timestamp and rather huge when calculating the elapsed time. These tests each run one million iterations, so the time for one iteration isn't huge and this doesn't matter much with smallish test/task runs. With bigger runs containing e.g. loops times start to add, though, because we call get_timestamp twice and get_elapsed_time once for each keyword, loop iteration, IF/ELSE branch, and so on.

What exactly we need to change to take datetime objects into use is still a bit unclear. One thing to consider is should we preserve existing starttime and endtime attributes all result model objects as they are for backwards compatibility. In practice we could store actual datetime objects to other properties (possibly under some new parent object like times that could be used like test.times.start) and the old attributes would be turned to properties.

If someone wonders why we have even created our own timestamp handling functions and didn't use datetime to begin with, the reason is that earliest Robot Framework releases supported Python 2.2 and datetime was added in Python 2.3. After we had added our own utils, switching didn't anymore make sense. I also got a feeling `datetime´ didn't have as good performance back then as it has now and at least it didn't have all the current features.

Thought about the API bit more and think this would work:

Model objects get new start_time and end_time attributes that contain the actual datetime objects (and are None by default).
Model objects also get new elapsed_time attribute that is property returning self.end_time - self.start_time. If either start or end time is not defined, should behave same way as nowadays.
Old starttime, endtime and elapsedtime are preserved as deprecated propertys that get the actual value from the aforementioned new attributes. They'd return same values as these attributes return currently.
For consistency, arguments passed to model objects when they are created are renamed from starttime and endtime to start_time and end_time. This is backwards incompatible, but creating these model objects ought to be pretty rare. They are more often only inspected by custom tools.

We probably should change the formatted timestamps from the current 20220308 16:36:12.123 format to more ISO 8601 compatible format like 2022-03-08 16:36:12.123. We should use it in logs, reports and in output.xml. In the latter we could consider using the official T as the separator between date and time parts, but with logs and reports it's better to use the space character.
The benefits of using the ISO 8601 format is to make the date part easier to interpret for humans and to make dates in output.xml easier to parse for external tools. Very importantly, we could use datetime.isoformat and datetime.fromisoformat that are very fast compared to using datetime.strftime and datetime.strptime we needed to use with custom times:
>>> d = datetime.now()
>>> d.isoformat(' ') == d.strftime('%Y-%m-%d %H:%M:%S.%f')
True
>>> timeit(lambda: d.isoformat(' '))
0.6668484999972861
>>> timeit(lambda: d.strftime('%Y-%m-%d %H:%M:%S.%f'))
1.6757996669912245
>>>
>>> s = d.isoformat(' ')
>>> datetime.fromisoformat(s) == datetime.strptime(s, '%Y-%m-%d %H:%M:%S.%f')
True
>>> timeit(lambda: datetime.fromisoformat(s))
0.1304736560123274
>>> timeit(lambda: datetime.strptime(s, '%Y-%m-%d %H:%M:%S.%f'))
5.866141628997866

A drawback of using 2022-03-08 instead of 20220308, and the reason we've used the latter so far, is that it's two characters wider. This probably requires us to adjust styles in log and report, but that's a small task and in general the benefits above are bigger than problems. The deprecated starttime and `endtime´ properties, discussed in the above comment, should nevertheless return the timestamps in the old format.

One smallish thing to decide is the timestamp precision. So far we have used milliseconds but datetime.now() uses microsecond precision. Due to Robot's own overhead, I got a feeling microsecond accuracy isn't that precise and would in general prefer using milliseconds.
We should use milliseconds at least in log and report to avoid timestamps getting overly long and, more importantly, to avoid adding three digits for each timestamp. Those added digits would probably increase log.html size considerably.
In output.xml we could still use microseconds if we wanted to. That would have a benefit that when we read the timestamp from there we get the same data as we originally had. If we only store timestamps in millisecond accuracy, then the data after the round-trip is different than what we originally for from datetime.now().
>>> d = datetime.now()
>>> s = d.isoformat()
>>> d == datetime.fromisoformat(s)
True
>>> d.isoformat(' ')[:-3]
'2022-03-08 17:36:38.972'
>>> s = d.isoformat(' ')[:-3]
>>> d == datetime.fromisoformat(s)
False

A drawback of using microseconds in output.xml is the increased file size. One way to reduce it would be that instead of storing start and end times in output.xml, we'd save start and elapsed time. The elapsed time in microsecond accuracy would look like 0:01:30.123456 and would be certainly shorter than end timestamp in millisecond accuracy. This obviously would be a backwards incompatible change, but if we are anyway changing timestamp format in output.xml this could be fine as well. Often the elapsed time is anyway more interesting to actual start/end times and in those cases having the elapsed time directly available would be convenient.  Anyway, if we decide to do this, the change should get its own issue.
If we decide to use milliseconds in output.xml, then we probably should use milliseconds also with the initial start/end times we get during execution. That way model objects would have same start/end time regardless are they created during execution or based on output.xml. That would mean that instead of just using datetime.now(), we needed to use something like this:
def now():
    d = datetime.now()
    ms = round(d.microsecond, -3)
    if ms == 1_000_000:
        return d.replace(microsecond=0) + timedelta(seconds=1)
    return d.replace(microsecond=ms)
...

The above obviously isn't as fast as using `datetime.now()` alone would be. In fact, it's even a bit slower than using the current `get_time_stamp_is`:
```python
>>> timeit(lambda: datetime.now())
0.2810397749999538
>>> timeit(lambda: now())
1.3667088740039617
>>> timeit(lambda: get_timestamp())
1.0332882049988257

Calculating the elapsed time would still be a lot faster than earlier, so even with this approach we'd get performance benefits overall. Anyway, either using microseconds in output.xml or just accepting that model objects have different times depending on do we get them during execution or from output.xml is probably a better idea. I think using microseconds in output.xml is a better idea.

Due to changes needed to output.xml, and output.xml being such an important interface between external systems, it's better to do this in RF 6.0.
A benefit of waiting for RF 6.0 is that it won't anymore support Python 3.6 that doesn't have datetime.fromisoformat.

This will be done in RF 7.0. For forward compatibility, it would be good to add propertys containing start, end and elapsed times as datetime and timedelta objects already in RF 6.1. I'll submit a separate issue about that.

I commented timestamp precision above and explained how rounding microseconds used by datetime.now() to milliseconds we currently use slows getting the timestamp down. I just noticed that datetime.isoformat has an optional timespec argument that makes it easy to round to milliseconds:
>>> dt = datetime.now()
>>> dt.isoformat(' ')
'2023-08-26 01:19:26.574297'
>>> dt.isoformat(' ', 'milliseconds')
'2023-08-26 01:19:26.574'

The performance with and withouttimespec is the same so it avoids performance issues. There would, however, be still the problem that times would have higher precision during execution (microseconds) than afterwards (milliseconds). My current thinking is that we should use microseconds in output.xml and decide what precision to use in the log file later.

In addition to changing start and end times of the result objects to datetime, we should change the timestamp of Message objects too.

This ought to be now done. Based on my quick tests performance benefits didn't really materialize, but there's a small boost in execution and processing output.xml with Rebot can be up to 10% faster. There are, however, various other benefits with these changes:

datetime is much easier to work with than our custom string timestamp.
Also timedelta has benefits compared to elapsed time being an integer representing milliseconds.
Timestamps in output.xml, syslog, and debug file now use the standard ISO 8601 format. That's especially nice with output.xml because the standard format is easier for external tools to process. Also in our schema we can now use xs:dateTime instead of xs:string that accepts anything.
We can easily use the ISO 8601 format with future outputs such as JSON results (#4847).
Attribute names start_time, end_time and elapsed_time are consistent with Python coding guidelines.
We now have elapsed time directly available in output.xml. External tools processing them doesn't need to calculate it based on the start time and the end time. The end time isn't there anymore so it needs to be calculated if needed, but I believe it's not often used than the elapsed time.
We store timestamps in microsecond accuracy in our result and in output.xml.
output.xml got a bit smaller because instead of starttime="20230907 20:00:01.000" endtime="20230907 20:00:11.123" we nowadays use start="2023-09-07 20:00:01.000000" elapsed="10.123000".

As discussed above, this issue caused some backwards incompatible changes and deprecations. I try to summarize them here:

output.xml now has start and elapsed attributes instead of starttime and endtime it used to have. Tools processing output.xml files themselves and interested in times need to be updated.
Related to the above, start now contains the timestamp in the ISO 8601 format like start="2023-09-07T20:00:01.000001" when we earlier had a custom format like starttime="20230907 20:00:01.000". elapsed is a float with up to six digits and it represents seconds.
Still related to the above, timestamp attribute with <msg> elements has been changed to time. Its format has also been changed from our custom timestamp to ISO 8601.
Also the generated time in output.xml uses ISO 8601.
Result model objects now accept start_time and end_time arguments when they used to be starttime and endtime. This shouldn't matter much, because results objects are seldom created by external tools. Inspecting them like earlier still works the same way as earlier.
starttime, endtime and elapsedtime attributes of the result objects are considered deprecated. There is no actual deprecation warning yet, but such a warning will be added later and these attributes will eventually be removed.
Utility functions related to getting and processing timestamps have been deprecated. Using them causes a deprecation warning and these utils will be removed in RF 8.0.

In practice normal users shouldn't be affected. The biggest problem is that external tools processing output.xml files need to updated, but timestamps being in a standard format ought to be nice for them in the long run. Previous Robot  versions cannot process new output.xml files, but output.xml files created by previous Robot versions are handled by the new version.

Due to the changes being pretty badly backwards incompatible, we added --legacy-output option for generating outputs in same format as RF 6 and earlier in c3c51708730ec7363541c366d560672b8f3d298f. It has a bug with Rebot (#5012), but it will be fixed in RF 7.0.1.


BUG: Explicit parsing of AWS S3 endpoint results in inconsistent behavior for GET

Outline
We're using S3FS to connect to S3 servers and e.g. download data. In some cases the S3 server would be e.g. a MinIO S3 and in other cases an AWS S3 server. To facilitate this distinction, we typically explicitly parse the endpoint URL as below:
import s3fs

def main_func():
    fs = s3fs.S3FileSystem(
        key="XX",
        secret="XX",
        client_kwargs={"endpoint_url": "http://s3.eu-central-1.amazonaws.com",},
    )

    test = fs.open("tf5test/4823E77F/00000048/00000001-5FB65005.MF4")
    print(len(test.read()))

if __name__ == "__main__":
    main_func()
    pass

Expected behavior
When running the above, we would expect to see consistently the length of the object we're trying to open.
Actual behavior
When we use the above, we do indeed get the expected result on most systems - but on some systems we experience an issue where the GET request fails. Specifically, below are partial debug outputs from a working vs. non-working system from using the above code:
Working system
DEBUG botocore.endpoint _do_get_response:140 - Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=http://s3.eu-central-1.amazonaws.com/tf5test/4823E77F/00000048/00000001-5FB65005.MF4, headers={'User-Agent': b'Botocore/1.17.43 Python/3.7.9 Windows/10', 'X-Amz-Date': b'20201120T103416Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIATDKWKZJD2AME4UXZ/20201120/eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=feb137f57fefc8743cf73b4790a9d3baef570922442d892cc4951dc6b3f4c83d'}>
DEBUG botocore.parsers parse:234 - Response headers: HTTPHeaderDict({'x-amz-id-2': 'qWolpseo/roMz33jonHXPgPxyjtGV5g4soxxWDHUNsJvZtN4GEza1BIJHvk7DSB7vEXVXE1Q+lU=', 'x-amz-request-id': '664EA4153E00EF73', 'date': 'Fri, 20 Nov 2020 10:34:17 GMT', 'last-modified': 'Thu, 19 Nov 2020 11:00:23 GMT', 'etag': '"0b7b628c7d29660d59a4aea328639e6c"', 'x-amz-meta-hw': '00.00', 'x-amz-meta-put-index': '2', 'x-amz-meta-fw': '01.03.01', 'x-amz-meta-ssid': 'WLAN-geschaeftlich', 'x-amz-meta-timestamp': '20201118T051030', 'accept-ranges': 'bytes', 'content-type': 'application/octet-stream', 'content-length': '40233854', 'server': 'AmazonS3'})

Non-working system
DEBUG botocore.endpoint _do_get_response:187 - Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=http://s3.eu-central-1.amazonaws.com/tf5test/4823E77F/00000048/00000001-5FB65005.MF4, headers={'Range': b'bytes=0-40233853', 'User-Agent': b'Botocore/1.17.43 Python/3.7.9 Windows/10', 'X-Amz-Date': b'20201119T133342Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIATDKWKZJD2AME4UXZ/20201119/eu-central-1/s3/aws4_request, SignedHeaders=host;range;x-amz-content-sha256;x-amz-date, Signature=b73e8ebb93927169b13565a8cbacfde483d6b8ab73903b32bd16826f05135547'}>
DEBUG botocore.parsers parse:234 - Response headers: {'x-amz-id-2': 'Q8qx6bLnj4+Z+D770bfAG7C1c+lNfWxkUvGj0FLKTFzNqIHNLeSqf1LUVXvw1nd0899/qQBC4dw=', 'x-amz-request-id': 'E5D8281755AA613E', 'Date': 'Thu, 19 Nov 2020 13:33:45 GMT', 'Last-Modified': 'Thu, 19 Nov 2020 11:00:23 GMT', 'ETag': '"0b7b628c7d29660d59a4aea328639e6c"', 'x-amz-meta-hw': '00.00', 'x-amz-meta-put-index': '2', 'x-amz-meta-fw': '01.03.01', 'x-amz-meta-ssid': 'WLAN-geschaeftlich', 'x-amz-meta-timestamp': '20201118T051030', 'Accept-Ranges': 'none', 'Content-Type': 'application/octet-stream', 'Content-Length': '0', 'Server': 'AmazonS3', 'Connection': 'keep-alive'}

As evident, the non-working system produces a different debug output and as part of it a 'Content-Length: '0', resulting in an error so that the file cannot be downloaded. Both systems are Windows 10, Python 3.7.9 and with the below pip freeze running in virtual environments.
If we remove the explicit parsing of the S3 endpoint, however, it seems to work - in the sense that both systems are able to correctly download the file. However, we struggle to understand the logic of this and we're hence concerned if the error may reoccur. Any help is appreciated!

pip freeze
aiobotocore==1.1.2
aiohttp==3.7.3
aioitertools==0.7.1
asammdf==5.23.1
async-timeout==3.0.1
attrs==19.3.0
bitstruct==8.11.0
botocore==1.17.44
can-decoder==0.0.2
canedge-browser==0.0.4
canmatrix==0.9.2
cchardet==2.1.5
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
docutils==0.15.2
fsspec==0.8.4
future==0.18.2
idna==2.10
influxdb-client==1.10.0
jmespath==0.10.0
lxml==4.6.1
lz4==3.1.1
mdf-iter==0.0.2
multidict==5.0.2
natsort==7.0.1
numexpr==2.7.1
numpy==1.19.1
pandas==1.1.0
pathlib2==2.3.5
python-dateutil==2.8.1
pytz==2020.1
PyYAML==5.3.1
Rx==3.1.1
s3fs==0.5.1
six==1.15.0
typing-extensions==<IP_ADDRESS>
urllib3==1.25.10
wrapt==1.12.1
xlrd==1.2.0
XlsxWriter==1.3.7
xlwt==1.3.0
yarl==1.6.3

s3fs certain is known to work for some non-aws s3 implementations.
Can you also turn on s3fs logging, so we know which call this is happening for? I see that the first block is a HEAD call and the second a GET call with bytes-range.
It may be a botocore error, or perhaps some setting that is needed (perhaps following a redirect?)

@MatinF , any further details here?

@MatinF , any further details here?

Hi Martin, we did not find further details/insight on this I'm afraid. We ended up with a solution that effectively removes the endpoint details when AWS is used.

Hi Martin, we did not find further details/insight on this I'm afraid. We ended up with a solution that effectively removes the endpoint details when AWS is used.


feature suggestion: import of packing list files

I've got a packing list in a text file. It would be nice if I could import it into packlist. (I don't want to type it in again, especially not a smartphone's virtual keyboard.)
(I've actually managed to import it, by finding the PREFS.xml, and exchanging its JSON payload with one I created with a few lines of javascript from the packing list text file I had. Nice hacking exercise, but not recommended for regular use...)

Good job...
Yes it is a bit complicated for future million of users :-)
Well storage in pref files is just a dumb easy to write  implementation implementing ISavingModule.
I plan to use a bit more efficient implementation when lists will begin to grow.
So I DO need an import functionnality...

Added, Will be available in incoming release 0.8

Hello code-surfer,
I have added most of what you suggested in release 0.8. (available since today 9th of april).
Please have a look.
Best regards, Nicolas

just discovering the app - it's great!
i am wondering if this could be extended to export from a website or a specific URL?
i use the following list as the mother of all packing lists for my trips: https://anarc.at/pleinair/liste/
it's more or less markdown in the back: http://src.anarc.at/wikis/anarc.at.git/blob_plain/HEAD:/pleinair/liste.mdwn
my process, usually, is to jot down items from that mega list on a piece of paper, then check the items as I pack them. it'd be interesting to reproduce that workflow here!

Hello @anarcat
I copied paste from the website you suggested into mass import screen of packlist, worked fine :-)
I start working on pre-saved list functionality that you can add or not as a whole to your next trip depending on what you are planning to do.
But may be add item by item and not only full list ??
See issue #8

yes, #8 covers part of what i am looking for: if the headers could be parsed to make categories, that would be great.
my idea of such an interface would be that there would be the "great list" and i would "swipe" things off of it that i don't need to make a list for the event that i need.

ah, and of course, if the app could talk directly through HTTP that would be great. not sure how i would go around copy-pasting a whole page like what you did...

After f-droid still didn't offer me the update (it's set to update daily), I ran an "update repos" manually and got the 0.8.
I think this new import feature is a good start! I simply did a "select all" in an email, pasted it into the packlist form and it worked.
It's just that people who'd like to use the item weight feature can't get them from this form (as it's currently implemented).
I wonder what the typical use case would be.
Before you implemented it, I wondered about a simple web page where people could enter their items, perhaps change their order and then download the list from the packlist app.
And of course upload, modify, download.
So far, I haven't had weights in my lists, but I like the idea.

Helllo @code-surfer do you think it would be useful to add a text format supporting an optional weight like this

books (500)
socks (50)
raincoat (850)
trousers

I'm not really the right one to ask -- I can easily get any format I prefer into any plain text format you chose. :-)
I'm not sure what less technically inclined people are likely to prefer. I guess many will have existing packing lists in spreadsheets. (Unlike most people I hardly ever use such programs.) Looking at LibreOffice Calc's supported formats, the simplest one is probably gonna be CSV. But it may be too simple, not providing any obvious choice for categories (#8).
If you can easily support markdown, then turning any header into a category and any list item into a packlist item may be a good way to go. Use of a library would be recommended, as e.g. headers come in several forms:
# this is one

this is also a header
===

My favorite would be something explicit like JSON, but most people would probably have major trouble getting their packlists into this format... (one typo and a JSON parser will reject it)
{ "groups": [ { "category": "Camping", "items": [ { "name": "tent", "weight": 1500 }, { "name": "sleeping bag" } ] } ] }

We now have a quiet complete import of basic list or more complex format lists / trips in packlist.
I close this quiet old topic and we can open a new one for additional discussion if useful.
Thanks for your suggestions.


[Doc] Configuring custom login page per service provider in web.xml is not documented

In this document [1], it only mentions about customizing the login page for service providers by checking the value of relyingParty parameter. Need to improve the same mentioning how we can achieve this using web.xml by adding the routing as below.
<!--  Custom Page configurations -->
<!-- *************** Application specific configurations ********************** -->
<context-param>
    <param-name>ServiceProviderApp-/samlsso_login.do</param-name>
    <param-value>http://localhost:8080/customauthenticationendpoint/login.do</param-value>
</context-param>

[1] - https://docs.wso2.com/display/IS570/Customizing+Login+Pages+for+Service+Providers

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Export issue in data field

whenever use select HTML element in any block and choose an option as selected,
when click on Export the Json result does not include any "date" field.
but when select an option manually the json rebuilds and this time include "data" field on selected item.
thank you

Hello @hamedf62
You use correct df- attribute?
Copy text of docs:
You can use the attribute df-* in inputs, textarea or select to synchronize with the node data.
Atrributs multiples parents support df--...
View https://github.com/jerosoler/Drawflow#node-example

Hello @hamedf62
Is it solved?

Closed due to inactivity


Add DTE D1.11 rate schedule

This PR adds the default D1.11 rate schedule from DTE, using the rates and fees published by the Michigan Public Service Commission's latest rate sheet publishing from DTE (archive.org for posterity.) These values include:

capacity charge
noncapacity charge
delivery charge
the Power Supply Cost Recovery (PSCR) charge (archive.org), which can change depending on the price of electricity DTE purchases but hasn't since Nov 2023.

I'm using this PR on my own HomeAssistant instance and it functions as expected. Let me know if you'd rather see this without PSCR and I'll update.
Sidenote: it's really neat to find an old colleague's personal code out in the wild, doing cool stuff :grin: I hope this message finds you well!

Thanks for the PR @rtertiaer! I hope you are doing well too!


their to no proper feedback form to it

Hello PA,
Their is no proper feedback form to it. I have added the feedback form to it and its in the pull request to mine.
Screenshot: before

Screenshot : After adding the feedback form

then

I have done the task its in the pull request of mine. I request you to assign the gssoc level to it. @sk66641
#106

@shivathmikavemula
Thank you for raising this issue. I've reviewed it, and it seems that no further action is needed at the moment. If you have any additional issues, feel free to reach out. 😊


[carbonmark] Pricing discrepancies

Prices aren't matching between /projects and /projects/[id]
Consider:

Are we reflecting the correct "Default pool price" (1 project per pool i.e. 4 specific tokens have zero redemption fee)
default projects are here https://github.com/KlimaDAO/klimadao/blob/dfa39653bbb9a67ccd79c2e13152a3da28b8bec8/lib/utils/getDefaultProjects/index.ts#L17
Are we correctly sorting to show the lowest price (in cases where a project exists in multiple pools)

this is resolved by #2049


Firefox rendering input and button controls with rounded border by default on Android

Bug seen in Firefox 49.0.2 and 50.0 on Android 6.0.1. Does not happen on Windows or Linux desktop. This can be fixed in Webui by adding default border-radius: 0;

Added default border-radius: 0


Create a Primary Button Rounded

One need to make this component using HTML and CSS. I would suggest to use Tailwind Playgrounds to make things faster and quicker.
Here is a reference to the component.

After building the component please raise a PR with a screenshot of the component and add the component in Button/button_primary_rounded.html.
If you need to use any icon please use it from Hero Icons
Good luck.

hey @Rajdip019, please assign me this. I would like to work on this issue.

Go ahead @aabhas-sao

i would like to work on this issue

Closing this issue as it is solved. @aabhas-sao


2 liter threshold issues.

This threshold should probably weighed somehow.    For Karts, it constantly generates too pessimistic message:
else if (currentGameState.FuelData.FuelLeft - fuelToEnd <= 2)
{
audioPlayer.playMessage(new QueuedMessage(RaceTime.folderHalfWayHome, 0, this), 7);
audioPlayer.playMessage(new QueuedMessage(folderFuelWillBeTight, 0, this));
}

For Karts or other fuel efficient vehicles, "2 liter warning" is not appropriate, probably, turn that off.

We probably should turn "2 liters warning" in Qualification

Would appreciate your thoughts/ideas here.

Additionally, "5 minutes of fuel left" "half fuel used" are not required in qualification.  Maybe we should have option for fuel warnings in qualification?

regarding qual and practice warnings, perhaps it makes more sense to prevent all the fuel warnings if we exit the pits with only 2 or 3 laps worth of fuel (as this is almost certainly a hot lap in qual or a qual simulation in practice)

I like that idea 👍

Yes, I like it to. As this ticket is assigned to you, 'make it so' :P

Should be just a case of changing that hard coded 2 for averageUsePerLap - will sort this today

Thanks Jim!


chore(renovate): add monaco-editor to ignoreDeps

Summary
This PR tries to pin monaco-editor by excluding it for @kong-ui-public/expressions and @kong-ui-public/entities-routes via packageRules in renovate.json.
Why?
monaco-editor is currently only used for the Expressions routes support in @kong-ui-public/expressions and @kong-ui-public/entities-routes. We encountered several issues while consuming the latest monaco-editor in the host app during the development. To deliver the Expressions routes support in time, we decided to downgrade it to v0.21.3–which we used previously in Kong Manager to prevent the Renovate bot from bumping it.
We will try to update it to the latest possible minor release after Kong Gateway 3.7's release.

@mptap Sure. I've updated this PR to use packageRules to scope the rule to @kong-ui-public/expressions and @kong-ui-public/entities-routes only.
Meanwhile, I don't feel the CI failure in the recent Renovate bumping PR (https://github.com/Kong/konnect-ui-apps/pull/3018) was caused by this change. Instead, it was because the Expressions editor component in @kong-ui-public/entities-routes imports monaco-editor, but we made monaco-editor a peer dependency instead of bundling it. To fix this, I've pushed a commit (https://github.com/Kong/konnect-ui-apps/pull/3018/commits/c65e71509076678a6566649ccb55a7662706f55a) to the Renovate PR to add monaco-editor as external in the Rollup config, as it is optional for us as of now (we don't show/use the Expressions editor at the moment), and I also updated several type imports from @kong-ui-public/entities-routes.


what is the step of using models to caseball?

Hello,
I know if I want to train a model, I should first use tombo to resquiggle fast5 files. And then convert  them to numpy array.
But if I want to basecall a fast5 file, should I use tombo to resquiggle fast5 files? Because I think I should not use the ground truth reference gnome to act as the tombo reference. But the combo need a reference patameter.

oh I know!
just follow your readme, use python ./scripts/basecall_original.py  is ok


chore: 🤖 revert file name generate logic

Summary

Test Plan

It seems like there's some missing context here. And it would be awesome if there is a test plan for preventing similar issues in the future.

Done


Add pyahocorasick to Conda-Forge?

Hello and thanks for a really great and easy to use package!
I'm using pyahocorasick in a bioinformatics application and I'm very impressed with its ease of use and speed when searching for strings (kmers) in whole-genome sequencing data.
I was wondering if you would be okay with pyahocorasick being added to Conda-Forge so that more people and applications can use it.
Below is a meta.yaml recipe for Conda-Forge using their example recipe:
{% set name = "pyahocorasick" %}
{% set version = "1.1.6" %}
{% set md5sum = "152985bf49c16ba26667f1471d232276" %}

package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  fn: {{ name }}-{{ version }}.tar.gz
  url: https://pypi.io/packages/source/{{ name[0] }}/{{ name }}/{{ name }}-{{ version }}.tar.gz
  md5: {{ md5sum }}

build:
  noarch: python
  number: 0
  script: python setup.py install --single-version-externally-managed --record record.txt

requirements:
  build:
    - python
    - setuptools
    - toolchain
  run:
    - python

test:
  imports:
    - pyahocorasick

about:
  home: https://github.com/WojciechMula/pyahocorasick
  license: BSD-3-Clause
  license_family: BSD
  license_file: LICENSE
  summary: Python module (C extension and plain python) implementing Aho-Corasick algorithm
  description: |
    pyahocorasick is a fast and memory efficient library for exact or
    approximate multi-pattern string search meaning that you can find multiple
    key strings occurrences at once in some input text. The library provides
    an ahocorasick Python module that you can use as a plain dict-like Trie or
    convert a Trie to an automaton for efficient Aho-Corasick search.
    It is implemented in C and tested on Python 2.7 and 3.4+. It works on
    Linux, Mac and Windows.
    The license is BSD-3-clause. Some utilities, such as tests and the pure
    Python automaton are dedicated to the Public Domain.
  doc_url: http://pyahocorasick.readthedocs.io/
  dev_url: https://github.com/WojciechMula/pyahocorasick

extra:
  recipe-maintainers:
    - WojciechMula
    - peterk87

With the recipe-maintainers in the example recipe, it states:
# GitHub IDs for maintainers of the recipe.
# Always check with the people listed below if they are OK becoming maintainers of the recipe. (There will be spam!)

So if you'd like pyahocorasick to be in Conda-Forge, would you like to be listed as one of the recipe maintainers?
Thanks!

Hi Peter, it's great to hear that effort of many people has a positive impact on your work. :)
Of course, it's perfectly OK to add the module to Conda-Forge. However, I should be listed as a CF package maintainer -- I don't know CF at all, and won't help in case of any platform-related problems.

Okay great! So I'll only add myself as a recipe maintainer. Thanks again for the great work!

Thank you. :) How do CF users report problems? Is there a separate issue list, or an issue goes directly to module's github page?

I think if they have issues with the CF build, then they could post their issues to the CF feedstock/recipe repo (https://github.com/conda-forge/pyahocorasick-feedstock), but if they have issues with the module, then they'll post them here.
The nice thing about CF is that you can install the pre-compiled module (and any dependencies it might have) without having to worry about your version of GCC being out of date. So I guess if you have a user that's complaining about not being able to compile pyahocorasick on their machine, you could point them at Conda and Conda-Forge.


Error:  Platform subaru does not generate unique IDs.

Just for background: Running in docker on pi4, I had tried to pull the latest HA core version, 2024.6.  After pulling, stopping and restarting the HA instance, something blew up.  Kept getting the error "connection refused."  Although I could log in, it kept "losing connection" and would never fully load.
Managed to revert to 2024.5.5 and everything magically worked EXCEPT the My Subaru instance.  So, I redownloaded the latest update folder, copied it over to the custom components, and it too started working.
Using "sudo cp -r /home/pi/Downloads/homeassistant-subaru-main/custom_components/subaru /home/pi/HomeAssistant/custom_components" to copy the extracted zip folder
The hitch is I get the below error in the logs whenever I reboot.  The "{Redacted}" parts are long strings that are all identical.  Note:  I have two vehicles in the app.  The Outback is the only one generating the errors.  The Crosstrek has no errors.
So, everything appears to be working.  But wondering if I need to do something about the errors below.
_Logger: homeassistant.components.binary_sensor
Source: helpers/entity_platform.py:744
integration: Binary sensor (documentation, issues)
First occurred: 7:20:32 PM (7 occurrences)
Last logged: 7:20:32 PM
Platform subaru does not generate unique IDs. ID {Redacted}_DOOR_ENGINE_HOOD_POSITION already exists - ignoring binary_sensor.2022_outback_touring_xt_hood
Platform subaru does not generate unique IDs. ID {Redacted}_DOOR_FRONT_LEFT_POSITION already exists - ignoring binary_sensor.2022_outback_touring_xt_front_left_door
Platform subaru does not generate unique IDs. ID {Redacted}_DOOR_FRONT_RIGHT_POSITION already exists - ignoring binary_sensor.2022_outback_touring_xt_front_right_door
Platform subaru does not generate unique IDs. ID {Redacted}_DOOR_REAR_LEFT_POSITION already exists - ignoring binary_sensor.2022_outback_touring_xt_rear_left_door
Platform subaru does not generate unique IDs. ID {Redacted}DOOR_REAR_RIGHT_POSITION already exists - ignoring binary_sensor.2022_outback_touring_xt_rear_right_door

Are the redacted strings the VIN for your Outback? Is it the correct VIN, or is it erroneously using your Crosstrek's VIN?
Please attach your device diagnostics output for both vehicles.

It is the VIN.  But, it is the correct VIN for the Outback.
subaru-fb6d9ee389174cdec9165d30101e6af4-2022 Crosstrek Limited-66cf7e2ff241b4ecebe5a75400824a50.json
subaru-fb6d9ee389174cdec9165d30101e6af4-2022 Outback Touring XT-0cf66a3fbe4c6fabd76208accf3fd641.json

Thanks. And just to confirm, everything is functioning correctly?

Appears to be.  I would have to go start the car manually to do anymore retests.  It doesn't like so many commands without actually starting it with the key.  but, I am able to locate, lock doors, etc.  Pressures, odometer,  and door sensors are reading correctly.

Just popping back on to add:  I had reverted to 2024.5.5 due to the startup loop issue I was having.  There were issues posted in the  HA Core Github around HACS causing a block on startup.  After reading that the issue had been addressed, I backed everything up, and then updated to 2024.6.1.
This did  allow me to run the latest stable version of core again.  However, I am still getting the same unique ID issue.  Was hoping the update might resolve the issue.  But, it did not.

Thanks. I'll try to figure out what's causing the error.

Everything IS still working.  I actually tested all the functions over the weekend and all of it works.  Just the odd error message.  Is it possible that something got duplicated when the HA core was acting up?  Any way to tell in the config files?

Maybe there are some old orphaned entities? If you view all your subaru entities, are there any duplicates or broken ones?  Your Outback should support reporting window open/close state. Is that working properly?

No duplicates showing in the graphical.  However, if I look in the PY cache folder, I am seeing duplicates with the exception of a number identifier on the end.  Everything is in there twice.  Once each with a "-311" suffix on the file name before the .pyc file type and another with a "-312" suffix.  All of the "-311" entries show a last modified date of 11/30/2023.  All of the "-312" files show a modify date of 6/5/2024 -which is coincidentally, the date that I had issues. Also of note, the timestamp appears close to the time I I recovered from the crash.  It Crashed at 17:37 (according to the dump) and I got it running on the old 2024.5.5 version a little over an hour and a half later after giving up on the stable 2024.6.0.  Time stamp for the duplicate files is 18:55.  According to the .fault file, all of the issues appeared to be python issues (consistent with the issues listed on the 2024.6.0 core github page).

I don't think any of the .pyc files would cause the error, since only the respective version would be used at runtime. If you've had the same HA instance over time, I'd expect to see .pyc files for both python 3.11 and 3.12.
Duplicates/orphaned entities that might cause the error would be found in config/.storage/core.entity_registry
Good to hear the windows status is working without error. I don't know why only a subset of subaru binary sensors are causing the duplicate entities error.
I can't find anything in the subaru integration that would cause this, as it does generate unique IDs formatted as VIN_<API_FIELD_NAME>. The HA core code that checks for collisions is here: https://github.com/home-assistant/core/blob/f02383e10db401a73927de5176ee436af8d533b6/homeassistant/helpers/entity_platform.py#L750

How many entries are returned when you run:
$ grep DOOR_FRONT_LEFT_POSITION config/.storage/core.entity_registry

There should only be two (one for each car).

Two results

that looks fine. the config_entry_id is going to be the same for everything associated with the subaru integration entity (i.e. your MySubaru account configured in HA).
If you were to grep for that id, it should list all entities for the subaru integration.

I did find a bug in my code that could possibly cause the issue for those with more than one vehicle. Please try 9a08a76

That did it


Process data after manually downloaded

I already download the data with the scripts in https://github.com/epic-kitchens/download-scripts
However, I could not find how to extra these .tars and make them gulp dataset that could be read by the epic-kitchens lib.
I've tried to read the Snakemake file, yet the download speed is a bit slow.
How could I use snakemake to create the same results as snakemake -p gulp_all with downloaded .tars? The downloaded files are int /tmp/epic_kitchen and I've changed the tmp dir in config.yaml

Hi there,
If you've downloaded the RGB and flow tars, you should be able to put them in the location snakemake is expecting them to be present (data/raw/$modality/$participant/$participant_video.tar, e.g. data/raw/rgb/P01/P01_01.tar), you might in addition have to 'touch' them so snakemake registers their existance:
# For both RGB and flow frames
$ snakemake -t download_all_frames

# Or for just RGB
$ snakemake -t download_all_rgb_frames

# Or for just flow
$ snakemake -t download_all_flow_frames

(see https://bitbucket.org/snakemake/snakemake/src/e049f5e49635ff9c56544a7b1efb5e1a193e1530/snakemake/init.py?at=master&fileviewer=file-view-default#init.py-776) or snakemake --help
Once you've done that you should be able to go ahead and run snakemake -p gulp_all


Cellular automaton

A simple simulation of the life cycle of cells. Watch populations evolve over generations.

👍


[UI] Select Pattern Component dropdown opening downwards now

Signed-off-by: Harshit Dandriyal<EMAIL_ADDRESS>Description
Select Pattern Component dropdown opening downwards now.

This PR fixes #5187
Notes for Reviewers
Signed commits

[x] Yes, I signed my commits.

@Chadha93 Review please

@Harshit-Dandriyal did you tried Popover to set the position? It may work without defining the static size, let's try https://v4.mui.com/components/popover/#popover

@Chadha93 popover helps in changing the position of opening of select, padding is done for internal styling of menuItem

@Harshit-Dandriyal did you tried Popover to set the position? It may work without defining the static size, let's try v4.mui.com/components/popover/#popover

@Chadha93 popover helps in changing the position of opening of select, padding is done for internal styling of menuItem

Gotcha!! you applied those styles to the menu. 👍


feat(http-log): add a feature

Summary

Checklist

[ ] The Pull Request has tests
[ ] A changelog file has been added to CHANGELOG/unreleased/kong or adding skip-changelog label on PR if unnecessary. README.md (Please ping @vm-001 if you need help)
[ ] There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Full changelog

[Implement ...]

message:
type:
scope:
prs:
issues:

Changelog
message: add a feature Introduce a new fields a and b and c
type:
scope: Plugin
prs:
issues:
- 1000

Issue reference

FTI-1000 KAG-100

/update-changelog


Remove aggregate and post transform from strategies

To simplify, we require all strategies to return a tensor of shape {:auto, n_trees, n_classes}.
We are currently removing custom forward and custom aggregate,
but I don't believe we have a use case for them. If we want to add
them back, it should be easy to handle them in the compile function
(like we do for post_transform).
PS: Tests pass but I didn't run the notebook.

Just got a chance to test this against the livebook. Just had to make a small change to TreeTraversal but otherwise everything works so im pulling in. Thanks again @josevalim for the PR and @polvalente for reviews


How to modify the Deploy Script?

Ok. This is actually two questions.

I want to run a custom shell script after I deploy my sources to the IFS folder. Where can I append/modify my shell script?

I also want to run a custom shell script after I made a successful connection to an IBMi server. (such as changing library list, setting up current library, changing CCSID and changing authority of users etc.,).

Needless to say, I would be using system command in Bash/QShell to execute the CL commands in both the cases above.

Create an action (in the actions.json file) that runs a specific command: https://halcyon-tech.github.io/docs/#/pages/developing/local/getting-started?id=_4-workspace-actions-deploy-amp-build

Sadly you can’t do that and altering the library list and such won’t affect the connection as we run most commands in a stateless manner. You likely want to check out profiles: https://halcyon-tech.github.io/docs/#/pages/settings/profiles


feat(config): support HTTP/2 via HTTPS_HTTP2_ENABLED env variable

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

3 of 3 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.0009%) to 99.864%

Totals

Change from base Build<PHONE_NUMBER>:
0.0009%

Covered Lines:
549

Relevant Lines:
550

💛  - Coveralls


Feature-request

Hi there, nice little library you've got there!
It would be nice if it would be possible to change the default options for the toastr notification (by calling the service), so we don't have to do it every time we call the "AddToastMessage".
Thanks for your work!

@os1r1s110 Definitely. I have not been able to put some time on it. I understand what you mean. We can add default settings at the time of service registration at the start up of the application and the library can use it through out the application. I will try to implement this as soon as possible. Thanks

done in commit 008ecf1
Please let me know if it is not working

Nice, I'll take a look and tell you if I see something wrong! Thanks for this super quick reaction and (IMO) really nice feature added.

Not a problem at all. Cheers.


Error updating

What model of system are you using?

[ ] Old 3DS
[ ] Old 3DS XL
[ ] 2DS (non-XL)
[ ] New 3DS
[ ] New 3DS XL
[check ] New 2DS XL

What is your current Luma3DS version?
If you are running an hourly version of Luma3DS, please manually update to a stable version as I will no longer deal with people who are using hourlies.

[ ] Luma3DS v7.1 + boot9strap v1.0/v1.1
[ ] Luma3DS v8.0 + boot9strap v1.2+
[ ] Luma3DS v8.1 + boot9strap v1.2+
[ ] Luma3DS v8.1.1 + boot9strap v1.2+
[ ] Luma3DS v9.0 + boot9strap v1.2+
[ ] Luma3DS v9.1 + boot9strap v1.2+
[check ] Luma3DS v10.0+ + boot9strap v1.2+

What version of Luma Updater are you using?

[ ] v2.0
[ ] v2.1
[ ] v2.1.1
[ ] v2.1.2
[ ] v2.2
[check ] v2.3

How are you running Luma Updater?

[check ] .cia
[ ] .3dsx + HBL/Rosalina

What problems are you experiencing?
Please tell us what problems you are facing.
What is the exact error code you are receiving?
Please give the exact error code the app gives you.
"Could not get location header for 3xx reply"
Ret Code: ded92
Steps to reproduce

v2.5 fixes this. Please manually update.

ok thank you

how do i do that?

Install the cia provided.

ok


cmd/server: start mocking account and customer routes

Issue: https://github.com/moov-io/gl/issues/6

Codecov Report

Merging #15 into master will increase coverage by 26.21%.
The diff coverage is 85.36%.

@@             Coverage Diff             @@
##           master      #15       +/-   ##
===========================================
+ Coverage   32.92%   59.14%   +26.21%
===========================================
  Files           4        4
  Lines          82      164       +82
===========================================
+ Hits           27       97       +70
- Misses         52       59        +7
- Partials        3        8        +5

Impacted Files
Coverage Δ

cmd/server/customers.go
95.23% <100%> (+20.23%)
:arrow_up:

cmd/server/accounts.go
74.6% <75%> (+1.26%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 794473c...cd1cd22. Read the comment docs.


More race condition fixes

Addition to changes made by @MSalopek:

Fixing some more race conditions
Refining lock in listen loop to avoid locking while reading from socket.

Thanks ;)


Different results for single-sequence FASTA query than same sequene in multi-FASTA query

I searched for a particular ORF sequence against a given database, using ssearch36.
fasta36 -E 10 -f 10 -g 2 orf.fasta library.fasta 2
Then I searched for a multi-FASTA file containing many ORFs, including the original one, against the same database.
fasta36 -E 10 -f 10 -g 2 lots_of_orfs.fasta library.fasta 2
I get similar hits, but the hits have different bit scores and evalues, I think driven by the different statistics.  Is this a 'feature'?
Example outputs illustrating the problem, this result for a single ORF:
   539214 residues in  2631 sequences

Statistics:  Expectation_n fit: rho(ln(x))= 8.6726+/-0.00279; mu= -2.9805+/- 0.107
 mean_var=53.5146+/-12.132, 0's: 0 Z-trim(86.9): 21  B-trim: 21 in 1/42
 Lambda= 0.175323
 statistics sampled from 722 (728) to 722 sequences
Algorithm: FASTA (3.8 Nov 2011) [optimized]
Parameters: BL50 matrix (15:-5), open/ext: -10/-2
 ktup: 2, E-join: 1 (0.559), E-opt: 0.2 (0.277), width:  16
 Scan time:  0.030

This result for the same ORF as part of a list of searches:
   539214 residues in  2631 sequences

Statistics:  Expectation_n fit: rho(ln(x))= 8.7380+/-0.0029; mu= -3.1580+/- 0.110
 mean_var=64.3984+/-15.325, 0's: 0 Z-trim(91.3): 11  B-trim: 24 in 1/42
 Lambda= 0.159822
 statistics sampled from 725 (728) to 725 sequences
Algorithm: FASTA (3.8 Nov 2011) [optimized]
Parameters: BL50 matrix (15:-5), open/ext: -10/-2
 ktup: 2, E-join: 1 (0.778), E-opt: 0.2 (0.49), width:  16
 Scan time:  0.030

The statistical estimates provided by the FASTA programs (including SSEARCH) are determined empirically, by sampling up to 60,000 scores from the library searched.
Since your library only has 2631 sequences, the estimates are based on all the scores that were calculated.
But since your library has different sequences, the distribution of scores is slightly different, and thus the statistical estimates are slightly different.
If you look at the numbers after the "Statistics:" line, you see that the rho, mu, mean_var, and Lambda are all slightly different, reflecting the different sets of scores that were found in the two searches.  These parameters were determined by fitting the scores that were obtained, and are used to calculate the E()-value and bit score.
I do think  of it as a "feature", since the estimates reflect the properties of the database that was searched.
Bill Pearson
Begin forwarded message:
From: Jay Moore<EMAIL_ADDRESS>Subject: [wrpearson/fasta36] Different results for single-sequence FASTA query than same sequene in multi-FASTA query (Issue #57)
Date: October 18, 2023 at 9:06:56 AM MDT
To: wrpearson/fasta36<EMAIL_ADDRESS>Cc: Subscribed<EMAIL_ADDRESS>Reply-To: wrpearson/fasta36<EMAIL_ADDRESS>I searched for a particular ORF sequence against a given database, using ssearch36.
Then I searched for a multi-FASTA file containing many ORFs, including the original one, against the same database.
I get similar hits, but the hits have different bit scores and evalues, I think driven by the different statistics. Is this a 'feature'?
Example outputs illustrating the problem, this result for a single ORF:
539214 residues in  2631 sequences
Statistics:  Expectation_n fit: rho(ln(x))= 8.6726+/-0.00279; mu= -2.9805+/- 0.107
mean_var=53.5146+/-12.132, 0's: 0 Z-trim(86.9): 21  B-trim: 21 in 1/42
Lambda= 0.175323
statistics sampled from 722 (728) to 722 sequences
Algorithm: FASTA (3.8 Nov 2011) [optimized]
Parameters: BL50 matrix (15:-5), open/ext: -10/-2
ktup: 2, E-join: 1 (0.559), E-opt: 0.2 (0.277), width:  16
Scan time:  0.030
This result for the same ORF as part of a list of searches:
539214 residues in  2631 sequences
Statistics:  Expectation_n fit: rho(ln(x))= 8.7380+/-0.0029; mu= -3.1580+/- 0.110
mean_var=64.3984+/-15.325, 0's: 0 Z-trim(91.3): 11  B-trim: 24 in 1/42
Lambda= 0.159822
statistics sampled from 725 (728) to 725 sequences
Algorithm: FASTA (3.8 Nov 2011) [optimized]
Parameters: BL50 matrix (15:-5), open/ext: -10/-2
ktup: 2, E-join: 1 (0.778), E-opt: 0.2 (0.49), width:  16
Scan time:  0.030
—
Reply to this email directly, view it on GitHubhttps://github.com/wrpearson/fasta36/issues/57, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABQYNPYYQN6AVA5AZJ6UPBTX77WBBAVCNFSM6AAAAAA6FV6HWSVHI2DSMVQWIX3LMV43ASLTON2WKOZRHE2DSOJYGI3TKNI.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

Thanks for the super-quick reply.  My doubt was that both of my searches use exactly the same library, but one has a single query sequence and the other has multiple query sequences including the single sequence.  It seems that the statistics are not just dependant on the library, but on the query sequences - the same query sequence gets different statistics depending on which other query sequences it is submitted with, even though the library does not change.  Hpoe this is clear.

The statistics do depend on both the query and the library sequences, but searching the same library with a single query sequence or that query sequence, included in a library with other separate sequences, should produce the same results (subject to sampling the library, which only takes place with more than 60,000 sequences).
If a query sequence is embedded in another sequence, the the statistics will be different.  But if one search has 1 query A, and another search has 10 queries including A, then the results for A should be the same.  The statistics for each query in a multi-query search are calculated independently.
However, I just did a test and learned that I am mistaken -- I get slightly different results when the query sequence is part of a multi-query library.  I will look into it.
Bill Pearson
Begin forwarded message:
From: Jay Moore<EMAIL_ADDRESS>Subject: Re: [wrpearson/fasta36] Different results for single-sequence FASTA query than same sequene in multi-FASTA query (Issue #57)
Date: October 18, 2023 at 1:02:10 PM EDT
To: wrpearson/fasta36<EMAIL_ADDRESS>Cc: William Pearson<EMAIL_ADDRESS> Comment<EMAIL_ADDRESS>Reply-To: wrpearson/fasta36<EMAIL_ADDRESS>Thanks for the super-quick reply. My doubt was that both of my searches use exactly the same library, but one has a single query sequence and the other has multiple query sequences including the single sequence. It seems that the statistics are not just dependant on the library, but on the query sequences - the same query sequence gets different statistics depending on which other query sequences it is submitted with, even though the library does not change. Hpoe this is clear.
—
Reply to this email directly, view it on GitHubhttps://github.com/wrpearson/fasta36/issues/57#issuecomment-1768973925, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABQYNP3XUEIV43TUE23W5VDYAADRFAVCNFSM6AAAAAA6FV6HWSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRYHE3TGOJSGU.
You are receiving this because you commented.Message ID: @.***>


Fix culture for cobertura xml report

Fixes https://github.com/tonerdo/coverlet/issues/463
@boldt comes from Germany where culture uses comma as decimal separator(like Italy).
This PR fix conversion using InvariantCulture that is defined as
The invariant culture is culture-insensitive; it is associated with the English language but not with any country/region.
doc https://docs.microsoft.com/en-us/dotnet/api/system.globalization.cultureinfo.invariantculture?view=netcore-2.2
Found this useful guide on formats...english languages uses dot. https://docs.oracle.com/cd/E19455-01/806-0169/overview-9/index.html
cc: @tonerdo

@tonerdo remember to change the merge strategy to "squash merge" to make easy possible reverts.


Improve plot_qv_data in QV Fitter

Summary
Improve plot_qv_data method in QV Fitter
Details and comments
Fixes #488:

[x] Change Y axis label to "heavy output probability".
[x] Error bar should be 2-sigma (success confidence interval requirement) instead of 1-sigma.
[x] Add caps to error bars
[x] Different markers for ideal and exp.
[x] Return matplotlib.Figure like plot_histogram in terra.
[x] Set figure default title inside the method rather than setting outside like the tutorial and textbook chapter
[ ] Remove show_plt argument because it doesn't do anything: whether is True or False the figure shows.

Did not remove show_plt argument as it would break existing tutorials and textbook chapter. It's redundant but it doesn't harm.
Before:

After:

Suggest reviewer: @dcmckayibm

@ShellyGarion Do you have any idea how to fix the docs and tutorials tests errors? I don't understand the error message of docs test. And for tutorials test, I think the changes I made should not break the existing tutorial.

The changes of ax to ax1 breaks the plotter unless ax is passed in as an argument. If you don't ax1 is never defined and this method fails. This is what the tutorial job is actually showing you, it says this pretty explicitly: https://github.com/Qiskit/qiskit-ignis/pull/495/checks?check_run_id=1082575091#step:5:58

I change ax1 back to ax. My local pylinter suggests that ax is not a valid variable name because it doesn't follow the format [a-z][a-z][a-z0-9]. But apparently this is accepted by the test.

@mtreinish Doc test passed! The release note is added as well. Thank you so much!

LGTM, thanks for making the updates


chore: automation test with core bundle

What does this PR do?
Implement automation tests with core-bundle
What issues does this PR fix or reference?
@W-15634776@
Test run: https://github.com/CristiCanizales/salesforcedx-apex/actions/runs/9291882439 ✅

Also fix the broken tests please

Please follow this PR for testing forcedotcom/source-tracking#589, and also add the automation of release.

I see that PR was closed though. Why is that?

@CristiCanizales I will checkout a new one in source-tracking since the existing PR has too many conflicts of dependencies with the main branch. So the issue is from package.json and yarn.lock, the test workflow is good to be referenced.


Correct a flag added recently, fixing formatting of exceptions thrown in catch blocks.

I recently added a parameter to AutoCatchHandlers to help detection
of whether or not an exception can be caught in user code, to match
the -nonative output better. Unfortunately, I set it incorrectly in
one place; namely, the case where we're in a "catch" statement made
by the user. Flipping the flag the other way fixes the issue.

why this change in the x64 case only? I see x86 and others are still set to true.

The issue is actually a bit more complicated than I first thought, on top of having to fix it appropriately for other architectures. There's apparently some inaccuracies with the scoping of the autocatchhandler in this case that is the true cause of the inaccuracy.


Bug fix/6397 not able to select tree component with matching name

Fixing, 6397 not able to select tree component with matching name

as created another pull request for same changed, so closing this request


Inverse Primary color style missing

Describe the bug
As we can see in below screenshot that on M3 website there is a mention of color "Inverse Primary" in theme generated by plugin.
To Reproduce
Steps to reproduce the behavior:

Generate new theme from plugin and you can crosscheck that this color is not being generated as style.

Expected behavior
This color style should be generated with plugin.
Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Thank you for the feedback!

@rodydavis Please ping me once it's live in plugin, thank you.

@rodydavis i know it is there in M3 docs, but that colour is not being generated as figma style by figma theme generator plugin, so i have created this issue to solve this. So let me know when it will be solves. Thanks

Inverse primary and shadow added on the latest!

so is there a workaround? they dont seem to be there in the web tool

Hello @rodydavis,
I'm still not seeing inverse colours in figma Plugin nor in the web theme builder. Has this feature been abandoned and the inverse colors deprecated(no documentation on how to generate these seems available)?

Hi @bpappin,
So On a hunch I doubled checked the Figma plugin output and realized it does not generate a frame with the inverse colors for you to see BUT the colors are in the Figma color Pallet.

That is correct, it's not reflected on the diagram but they are available in styles.
The plug-in is not abandoned and is under active development.


Bump aws-sdk to 1.11.697

Fixes https://github.com/prestosql/presto/issues/2614 and https://github.com/prestosql/presto/issues/1132

supersedes https://github.com/prestosql/presto/pull/2971

Thanks!


Add color option to media player card

Hi @pandaboy6621 can you describe a little more your need ? :)

I don’t have an icon color picker for my media player card
On Fri, Apr 15, 2022 at 11:55 PM Pierre-Emmanuel Mercier <
@.***> wrote:

Hi @pandaboy6621 https://github.com/pandaboy6621 can you describe a
little more your need ? :)
—
Reply to this email directly, view it on GitHub
https://github.com/piitaya/lovelace-mushroom/issues/328#issuecomment-1100589431,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AP3U57LLL7ISHP4XCJ4X463VFJP7VANCNFSM5TRAZLCQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

Same here, it would be nice to be able to change the icon in the media player card. Also in yaml mode, it's not an option.

Agreed, it would be nice to have an option to change the color, just like with the entity card

Yeah, I find it strange that the default color is indigo. It should by default follow the default color scheme, with the option to choose another color.

Any update on this?


Expansion bug when braiding half-bubbles

Construct a half-braid on two half-bubbles:

Now click on the central point and drag left. This trips an assertion.

expansion_bug.zip

Very likely related to #926 but should be simpler to debug.


Add support for >100% in alert cpu load feature

Added in https://github.com/nozel-org/freebsd-serverbot/commit/9297b2545c296697b32938eb851bd90d3e5e12ce.


Clarification required

It took me a while to work out what the below code does (referenced above), after which I realised all it was for was the three column layout. It would be nice if you added an explination to this as it's fairly insignificant bearing in mind this is a data focussed tutorial. Until I'd worked it out, I thought it had a more important function.
@{
int cnt = 0;
List<ContosoUniversity.Models.SchoolViewModels.AssignedCourseData> courses = ViewBag.Courses;
                foreach (var course in courses)
                {
                    if (cnt++ % 3 == 0)
                    {
                        @:

                    }

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 938b178b-5f78-e673-df41-0409ad4412e1
Version Independent ID: c06d8343-1407-aa05-de96-c1aca71ba232
Content: Tutorial: Update related data - ASP.NET MVC with EF Core
Content Source: aspnetcore/data/ef-mvc/update-related-data.md
Product: aspnet-core
Technology: aspnetcore-data
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

That's part of programming, figuring out how code works.

Surely that's the whole point of a tutorial... with that logic, why not just forget the tutorial and just give me the code to look at?
All it requires is one line to explain the point of this section of code, as for those who are beginners, it can really help them out. I didn't ask for a condescending response, I was hoping you'd take this as constructive feedback... clearly not.

With a better title, this is very discoverable. Can you fix that? Also, edit and fix the code. You can't past code into the doc page, but you can fix it here.
There are hundreds of similar snippets in this series. It's not a beginning tutorial.

Any chance you could PR this? At the top right of the article, select the Edit pen icon:

Select the pen icon again. Save.


showing scale / axes visual in empty viewer on startup can segfault

🐛 Bug
It was noted in #1835 that showing a scale or axes visual (using the menubar) after launching napari from the command line with napari can trigger a segfault. Commenting out the text node in the welcome visual prevents it from happening.
(base) czirwc1macos2701:napari nsofroniew$ napari
WARNING: Error drawing visual <vispy.visuals.line.line._GLLineVisual object at 0x7faa2ba51588>
18:17:02 WARNING Error drawing visual <vispy.visuals.line.line._GLLineVisual object at 0x7faa2ba51588>
/Users/nsofroniew/opt/anaconda3/bin/pythonw: line 3: 63069 Segmentation fault: 11  /Users/nsofroniew/opt/anaconda3/python.app/Contents/MacOS/python "$@"

Once some visuals have been added to the viewer this doesn't seem to happen, even if the viewer is made empty again

@sofroniewn : similar to https://github.com/napari/napari/issues/1905, do you think this was also fixed by #2542?

yup - fixed, just tried it and no error - will close. thanks!!


Eroare multipla la orase job

ENVIRONMENT: production
URL: https://scrapers.peviitor.ro/src/3ss/index.html
Browser: Chrome
Device: Laptop
OS: Windows
STEPS TO REPRODUCE:
Deschide URL in browser
Apasa RUN SCRAPER
EXPECTED RESULTS:
la toate joburile apar orașe din România
ACTUAL RESULTS:
apare eroarea "BraÈov is not a city in Romania,Targu MureÈ is not a city in Romania,Cluj-Napoca"

Cities have been updated.


kenburns-dom standalone usage

When using kenburns-dom standalone, the start method
deviates from the kenburns module (runStart and run).
Also, runStart of kenburns-dom apparently also doesn't
return a promise or callback for the finish of the animation.

I've released a major version of the library (2.0.0) that have a new documentation. https://github.com/gre/kenburns/blob/master/API.md
checkout also the example: https://github.com/gre/kenburns/blob/master/example/index.js
You now only need to use the library kenburns and no more kenburns-dom.
To use the dom implementation:
import KenBurnsDOM from "kenburns/lib/DOM";
var div = document.createElement("div");
div.style.width = "400px";
div.style.height = "400px";
const kenBurnsDOM = new KenBurnsDOM(div);
// kenBurnsDOM.animate(...).then(...);
// kenBurnsDOM.animateStep(...);

animate(image, fromCrop, toCrop, duration/*, easing*/) runs the animation and returns a Promise.
if you want finer control, animateStep(image, fromCrop, toCrop, progress) allows to render one frame of the animation. It's lower level but you can do whatever speed / easing / interruption / ...


[Feature request] Add server url to "API keys and API docs" section

Is your feature request related to a problem? Please describe.
When linking lnbits with external services, (e.g. zaprite) one needs to specify two things: node url and invoice key.

Invoice key is clearly visible in the "API keys and API docs" section, but it's sometimes unclear what my "LNbits Node URL" is.

Describe the solution you'd like
Display "LNbits Node URL" in "Node URL, API keys and docs"

This is great. Would definitely help our users.


"Null products pasted" on an empty list

"Null products pasted" on an empty list

Should not happen again as we removed the concept of "paste".
Feel free to reopen the issue if needed.


Support for object literal property value shorthand

Started here: https://typescript.codeplex.com/workitem/1367
Please consider adding support for object literal property value shorthand, as per the ES6 proposal: http://wiki.ecmascript.org/doku.php?id=harmony:object_literals#object_literal_property_value_shorthand. This would allow for the following:
function foo(a: number) {
  var b = 123;
  return { a, b };  // instead of { a: a, b: b };
}

Fixed by #1127 and #1346

In my codebase this feature would very useful, because I'm often creating "smaller" object from "bigger" one, so for loop on "smaller" object enumerates just the properties I need.

You can use destructuring with object literal property value:
var bigObject = {
    A: "1",
    B: "2",
    C: "3"
};

var {A,B} = bigObject,
    smallObject = {A,B};

:rotating_light: untested code


[FEATURE REQUEST] Add badge number field for PD officer, FD/EMS card

Add badge number field for PD officer, FD/EMS card

Example of LAPD officer badge

Officer badge number has been implemented. EMS/FD will come once starting on EMS/FD dashboard.


I have Added link to the website in the website name

I have added link to the website in the website name. Please accept the PR for Hacktoberfest.

I have added the link in the Dev Profiles in both index.html and project.html. Please check it and accept the PR

I have added the link in the Dev Profiles in both index.html and project.html. Please check it and accept the PR

why are you not added in about page, make change and push the change


Is WorkerService SDK still in Beta?

Hi,
The link to the article is
https://docs.microsoft.com/en-us/azure/azure-monitor/app/console
The Service I am referring to is Azure Application Insights
Issue Description:
On top middle of the page there is a note indicate that there is a new beta Application Insights SDK called Microsoft.ApplicationInsights.WorkerService. But in other pages, I don't see anywhere that tells the SDK is still in Beta? And in nuget description it doesn't show Beta neither. Please help to verify if the SDK is still in Beta.
Thanks.
Best,
Eric

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b189efce-ce4e-f8f8-ecf8-8803a18f9b65
Version Independent ID: aec54012-dae3-ce18-0f8c-30749c46a79c
Content: Azure Application Insights for Console Applications
Content Source: articles/azure-monitor/app/console.md
Service: azure-monitor
Sub-service: application-insights
GitHub Login: @mrbullwinkle
Microsoft Alias: mbullwin

@ezheng2019 Thank you for your question. I will investigate and get back with you shortly.

The package is out of beta as you can see on the Nuget Gallery page for it: https://www.nuget.org/packages/Microsoft.ApplicationInsights.WorkerService
I'm assigning this to the doc author to handle appropriately.

Thanks again for reporting this. The doc has been updated internally, the change will go live between 3-4 PM today UTC-8.

#please-close


cargo insta test does not work for inline doc assert

What happened?
When trying to update snapshots located in doc comments, insta recognizes that the snapshot is incorrect, but fails to prompt for review and creates a .pending file.
Reproduction steps
Snippet:
/// ```rust
/// use insta::assert_debug_snapshot;
///
/// assert_debug_snapshot!("abc", @r###""###);
/// ```
#[derive(Debug)]
pub struct A;

cargo insta test -- --doc OR cargo test --doc
When cargo test --doc was run in src/ a new file was created: .lib.rs.pending-snap with the following contents:

{"run_id":"1660578077-420954000","line":5,"new":{"module_name":"rust_out","snapshot_name":"_doctest_main_src_lib_rs_2_0","metadata":{"source":"src/lib.rs","assertion_line":5,"expression":"\"abc\""},"snapshot":"\"abc\""},"old":{"module_name":"rust_out","metadata":{},"snapshot":""}}

cargo insta review

results in: done: no snapshots to review
Insta Version
insta v1.18.2
rustc Version
rustc 1.65.0-nightly (801821d15 2022-08-14)
What did you expect?
That insta would update the snapshot and prompt for review, or at least show a warning or note that such an operation isn't supported.
Is this behavior intended? What should I do instead? Thanks for creating such a valuable and great tool! I have used it in a few projects, and it's been great so far!

Unfortunately rust does not have correct line information within doctests (https://github.com/rust-lang/rust/issues/79417) which makes it impossible to reliably reproduce the original location. In theory the offset to the doctest can be calculated from the function name as it encodes currently the offset to the doctest there, however the internally added preamble is not known and not consistent.
I will add a warning and avoid emitting the pending snapshot though, that can be detected.

This warning is also incorrectly triggered when using cargo test instead of cargo insta test on tests that can actually be updated.

@konstin what's an example of that?

Closing as inactive but please reopen with a repro if available


Check if environment key is set before deleting in HyperAI orchestrator

Describe changes
This PR adds a check to see if ENV_ZENML_HYPERAI_RUN_ID is available in environment before attempting to delete it via del. I just tried scheduling a pipeline and ran into this issue.
Pre-requisites
Please ensure you have done the following:

[x] I have read the CONTRIBUTING.md document.
[x] If my change requires a change to docs, I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[x] I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
[x] If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Other (add details above)

Thanks, @christianversloot !


Hamburger button doesnt work

Hamburger onClick function is not present to expand navbar
I want to work on this issue

@ashuvssut I have kept this intentionally, as someone is already updating nav-bar in Issue #465
(As the nav-bar was making the Top Home Section height more than 100vh in mobile screen so I just hide the nav-links and added a hamburger icon, keeping in the mind that the one who is updating and making new nav-bar will fix this)

hey @sanjay270899, the assigned contributor in that issue seems to be a newbie in React. He will take some time to resolve the issue. I think I can give a fix for this (consider it as temporary)


[bug] Regression on 1.24.3 on CI environment

Describe the bug
I have a big graph when running moon ag :deploy. I can post it as a comment if you want.
In moon 1.24.2 when i run moon run :deploy in CI (Github) it works fine.
Now in 1.24.3, no task is being run. Error is:
No tasks found for target(s) :deploy

Locally in 1.24.3 moon run :deploy runs fine.

@rhuanbarreto I'm assuming these deploy tasks have runInCI: false enabled? Let me look into it.

All of them have local: true

@rhuanbarreto Can you try 1.24.4

Fixed! But we still have problems with proto. Will open a separate issue.


docs: add Render hosting guide

Motivation
Render is an official deployment option for create-react-app, gatsby, and hugo. This PR adds a guide for Docusaurus users who may want to host their site on Render.
Have you read the Contributing Guidelines on pull requests?
Yes.
Test Plan

[x] yarn prettier && yarn lint
[x] yarn test
[x] Visual inspection of all doc versions.

Makes sense and will submit an update shortly.
On Thu, Jul 11, 2019 at 9:38 PM Yangshun Tay<EMAIL_ADDRESS>wrote:

@yangshun requested changes on this pull request.
Sorry for the late review. The PR looks ok but can we leave out updating
the old versioned docs? The whole intention of versioning is to freeze old
versions and new stuff should only go into new versions.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/facebook/docusaurus/pull/1615?email_source=notifications&email_token=AAAOUANEX5XBSAVFHI5PD5TP7ADEZA5CNFSM4HZNGQA2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB6HWKCI#pullrequestreview-261055753,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAOUAJT5QPTW3PBONXY33TP7ADEZANCNFSM4HZNGQAQ
.

Sorry for the late review. The PR looks ok but can we leave out updating the old versioned docs? The whole intention of versioning is to freeze old versions and new stuff should only go into new versions.

@yangshun PR updated.


devicetree bindings: improve support for compatible compatibles

The st,lis12dh driver supports multiple variants: SPI and I2C buses, and at least two variant chips st,lis3dh and st,lis2dh12.  PR #19901 adds a reference to the latter.
In the original submission checkpatch complained about:
-:11: WARNING:UNDOCUMENTED_DT_STRING: DT compatible string "st,lis2dh12" appears un-documented -- check ./dts/bindings/
#11: FILE: boards/arm/nrf52_pca20020/nrf52_pca20020.dts:123:
+		compatible = "st,lis2dh12", "st,lis2dh";

It appears that to make this work we need to duplicate the YAML with the name of the alternative.  This means that the property descriptions are now replicated in four different binding files.
In #19624 it is proposed to add another property, which would have to be added to all those files.  In an attempt to simplify that I prepared https://github.com/pabigot/zephyr/commits/pr/20191017b where the top commit refactors the properties to a file that's included by all the others.
This results in the following build failure:
-- Overlaying /mnt/nordic/zp/zephyr/dts/common/common.dts
devicetree error: both /mnt/nordic/zp/zephyr/dts/bindings/sensor/st,lis2dh-i2c.yaml and /mnt/nordic/zp/zephyr/dts/bindings/sensor/st,lis2dh-spi.yaml have 'compatible: st,lis2dh'
CMake Error at /mnt/nordic/zp/zephyr/cmake/dts.cmake:177 (message):
  new extractor failed with return code: 1

It's not obvious to me how the Linux checkpatch is satisfied by having st,lis2dh12 appear as a compatible in a file st,lis2dh12-i2c.yaml but without that file it complains.  Nor is it obvious to me how putting the properties into a common include file causes breakage of the unique-compatible rule when duplicating them in separate files does not.
Nonetheless, having to replicate all properties in multiple binding files just to allow specification of variants to get past checkpatch is not a very maintainable solution.  Something should be done.

Looks like scripts/checkpatch.pl is just doing a recursive grep for the strings in dts/bindings/. Maybe it could be hacked around with a comment like
# Mention this compatible just to make checkpatch.pl happy
# st,lis2dh12

Would that get awkward?

I don't think it'd get awkward, but it should probably be handled by an explicit entry exposed in the YAML either by making the compatible  key accept a list, or adding an also-compatible key with non-primary entries.
I think we may have touched on this before somewhere, regarding the change in how compatible works in devicetree between the original text model and DTSchema.  Though in both cases the binding description is expected to have a list of acceptable compatibles.
If things like st,lis3dh-i2c.yaml exist only to satisfy this checkpatch requirement (which seems to be the case) I'd be in favor of moving to multiple compatibles in a single binding, however that gets done.

See https://github.com/zephyrproject-rtos/zephyr/pull/20289#issuecomment-554483726 for additional information.


MPI node shared memory

working to reduce memory usage in MPI jobs, see https://github.com/healpy/pysm/issues/30
Currently we read all input maps on MPI rank 0 from disk, then broadcast each of them to all MPI processes. A node could have up to 16 MPI processes for software that scales poorly with threads.
So we could have 16 copies of a map in a single node.
We want instead to allocate a node-shared buffer with MPI, then read the map on rank 0 of MPI Comm world and then have 1 proc per node receive the map and copy to the node-shared buffer. So we have just 1 map per node.

@keskitalo I think I got this working, can you please review the PR? and merge if you think it is fine.
I am still waiting for the Travis tests.

@keskitalo I have ran my usual test at 4096,

works fine with 1 and 2 nodes, 2 MPI processes per node.
works fine with 6 nodes, 6 MPI processes per node
but the full 13 nodes and 6 MPI processes per node hands for more than 20 minutes in the communication part.

working on more debugging

memory consumption is way lower in the run of 6 nodes x 6 MPI procs:
(pysm3) [zonca@popeye-login2 mpi_examples (master)]$ seff 71105
Job ID: 71105
Cluster: slurm_cluster
User/Group: zonca/zonca
State: COMPLETED (exit code 0)
Nodes: 6
Cores per node: 48
CPU Utilized: 13:30:17
CPU Efficiency: 43.96% of 1-06:43:12 core-walltime
Job Wall-clock time: 00:06:24
Memory Utilized: 570.56 GB (estimated maximum)
Memory Efficiency: 12.68% of 4.39 TB (750.00 GB/node)
(pysm3) [zonca@popeye-login2 mpi_examples (master)]$ seff 71107
Job ID: 71107
Cluster: slurm_cluster
User/Group: zonca/zonca
State: COMPLETED (exit code 0)
Nodes: 6
Cores per node: 48
CPU Utilized: 13:19:25
CPU Efficiency: 46.39% of 1-04:43:12 core-walltime
Job Wall-clock time: 00:05:59
Memory Utilized: 142.49 GB (estimated maximum)
Memory Efficiency: 3.17% of 4.39 TB (750.00 GB/node)

the 13 nodes example just hangs after reading the 3 maps for dust (each polarization is separate)...@tskisner any suggestion on how to debug a MPI job that just hangs forever?

I actually got the memory usage inverted previously, MPI Shared uses more memory.
Here is a test of 6 nodes, 6 MPI processes per node, mpi_shared:
Job ID: 71112
Cluster: slurm_cluster
User/Group: zonca/zonca
State: COMPLETED (exit code 0)
Nodes: 6
Cores per node: 48
CPU Utilized: 13:28:48
CPU Efficiency: 45.05% of 1-05:55:12 core-walltime
Job Wall-clock time: 00:06:14
Memory Utilized: 597.27 GB (estimated maximum)
Memory Efficiency: 13.27% of 4.39 TB (750.00 GB/node)

Same job on master:
Job ID: 71144
Cluster: slurm_cluster
User/Group: zonca/zonca
State: COMPLETED (exit code 0)
Nodes: 6
Cores per node: 48
CPU Utilized: 13:16:52
CPU Efficiency: 46.50% of 1-04:33:36 core-walltime
Job Wall-clock time: 00:05:57
Memory Utilized: 142.17 GB (estimated maximum)
Memory Efficiency: 3.16% of 4.39 TB (750.00 GB/node)

this is disappointing, I'll look again tomorrow

@keskitalo I simplified the code, I think it is correct now, but still I get high memory usage from seff:
Job ID: 71179
Cluster: slurm_cluster
User/Group: zonca/zonca
State: COMPLETED (exit code 0)
Nodes: 6
Cores per node: 48
CPU Utilized: 13:24:20
CPU Efficiency: 44.92% of 1-05:50:24 core-walltime
Job Wall-clock time: 00:06:13
Memory Utilized: 598.39 GB (estimated maximum)
Memory Efficiency: 13.30% of 4.39 TB (750.00 GB/node)

I wonder if this is maybe counting the MPI shared memory more than once.

Maybe you could call memreport inside the PySM operator at where you think the memory high water mark is?

thanks @keskitalo
I am still running PySM outside of TOAST to test the MPI shared implementation.
I tested it with a single model for dust (3 maps .77 GB each),
initial memory usage is ~9GB, with the mpi_shared implementation memory usage inside the PySM routines is max 10.3 GB, with master it is 14.7 GB.
I will merge this, please test if this is improving memory usage in the simulations.


Fix typos

Thanks a lot and sorry for the delay!

:partying_face: :partying_face: :partying_face: :partying_face: :partying_face:


Refactor code: Create a class for the Image-Manager

Closes #228.
Create a single class that contains the whole functionality of the
Image-Manager which allows for easy exporting.
The Image-Manager can be invoked by running the main() method.
All unittests are updated accordingly.
The requirements were also updated, which was forgottten in #236.
Signed-off-by: Simon Hensel<EMAIL_ADDRESS>

@ra-rau Please test this one.

(Context: This came up in SCS team 1 today and I was invited to comment here, so I hope this is helpful.)
TL;DR: If the goal is reusability, then having individual functions with clear arguments (as they were before) seems more sensible to me.
That way, the image manager becomes also a library which can be used, instead of a monolithic class with non-pure dependencies (such as the CONF) and passing state between methods implicitly (via attributes).
Many of the functions have been transformed from taking a "context" argument (the connection) plus the arguments describing the operation (image name etc.) into taking self + the arguments taking the operation. That is an indicator that they should be free functions which should just take the single context argument.
In other cases, some of the function arguments have been moved to object attributes, even though they are part of transient state, making the object a kind of "register" to store state (sorted_versions is one such thing). sorted_versions is only used by process_images and rename_images, which are directly called from main(). If sorted_versions was instead passed as argument to process_images, it could in fact be used by someone instantiating the class; otherwise, this attribute needs to be set beforehand which is weird.
If the functions are transformed back to only use arguments or inherent object state, they could easily be transformed back to functions (based on the argument above), removing the mental overhead of using a class (and its potentially non-obvious state).
Having purer functions (no global/shared state beyond what is passed as arguments and what is returned) has the following advantages:

Easier testing: no need to mock global state, passing mock objects as arguments is sufficient
Easier reuse: no need to ensure that some shared state is correct; the arguments of the functions are sufficient to make them work correctly.

@horazont Thanks for the comment, that's a good insight. I understand your point of view and agree with it. Pure, self-contained functions make code reuse and testing (slightly) easier and more straightforward.
The main advantage of a class is the reusability of class variables, which reduces the number or arguments a function has to take, since they can make use of a shared state (e.g. by moving the connection to self). But then, the connection should be set in the constructor instead of main(), so that it is available as soon as the object gets created.
In the case of sorted_versions, the usage is indeed awkward and could be improved.
If all function parameters are made consistent by using self + all arguments prescribing the operation, all functions work independent of each other (just like when using pure functions), since the shared state containing CONF and conn is established when the object is created.
In the end it comes down to which design principle is preferred, the code works just as well without using a class.


Use huggingface-embeddings to load local embedding model

Is your feature request related to a problem? Please describe.

I am using the following configuration and a downloaded model as I want to deploy on an instance with no internet access
backend: huggingface-embeddings
embeddings: true
name: all-minilm
parameters:
  model: all-MiniLM-L6-v2.bin

However I get the following error at inference :
{
  "error": {
    "code": 500,
    "message": "could not load model (no success): Unexpected err=RepositoryNotFoundError('401 Client Error. (Request ID: Root=1-64e750c4-1941428a25f0e1956ea10fa1;33620328-90bf-49a5-9e52-01fa29fa39d6)\\n\\nRepository Not Found for url: https://huggingface.co/api/models/sentence-transformers/all-MiniLM-L6-v2.bin.\\nPlease make sure you specified the correct `repo_id` and `repo_type`.\\nIf you are trying to access a private or gated repo, make sure you are authenticated.\\nInvalid username or password.'), type(err)=<class 'huggingface_hub.utils._errors.RepositoryNotFoundError'>",
    "type": ""
  }
}

I indeed specified a bin file, and my other models work well so it should in theory look into the correct folder.
Describe the solution you'd like

If there is a file extension load a local model, or add a parameter for that.

well - actually @localai-bot is correct here. the huggingface-embeddings backend wants a huggingface repository in the model name. See the documentation in https://localai.io/features/embeddings/index.html#huggingface-embeddings . moving to a discussion


front-end template

need something goodie using bootstrap, with widgets, etc.

Perhaps consider a JS framework to do the heavy lifting on the front-end. Here's a Laravel integration library for React.js: https://github.com/talyssonoc/react-laravel
It will increase the learning curve for new theme developers, but on the other hand there's quite a few free react themes floating around that can quickly be adapted.

I was thinking about doing that, but decided against it, simply because of the learning curve, not only for any theme devs, but also for me. Regardless, the API endpoints will be there for whoever wants to do that, though


Auto corrected by following Layout/TrailingWhitespace

Auto corrected by following Layout/TrailingWhitespace
Click here to configure it on awesomecode.io

@Br3nda why are you working on this repo?


PersistentKernel with Prefetch

Persistent Kernel for MI, support StoreRemap, FP16/BF16 HPA, WiderLocalRead.
PersistentKernelAlongBatch: Persistent Kernel WG can also distributed along batch dimension
Implement PrefetchAcrossPersistent

WIP, need to rebase with latest PR. And also need to resolve the HostLibraryTests data conflict.

@ramjana thanks for reviewing. I'll merge this version first and keep improving this as your suggestion. But it's about the code freeze date so I'll do it at the beginning of next release.


Fix and Enable All Staticcheck Checks besides ST1003

Adds a staticcheck.conf under v2/ to skip ST1003 since many identifiers violate the recommended practices. These include package names with underscores, variables names written in all caps, and uncapitalized initialisms such as Http.

Thanks for that!


todo: Generate PDF reports for exectives/telecaller for each day.

todo: Generate PDF reports for exectives/telecaller for each day.

done