Support for creation of symlinks in file API

Required information

Distribution: Ubuntu
Distribution version: 16.04
The output of "lxc info" or if that fails:

config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
api_status: stable
api_version: "1.0"
auth: trusted
public: false
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIFcTCCA1mgAwIBAgIRAKrvdFAi5uDuycC8KnphJ/QwDQYJKoZIhvcNAQELBQAw
    RDEcMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEkMCIGA1UEAwwbcm9vdEB0
    aW1zZXJ2ZXIudGltc3N0dWZmLmV1MB4XDTE3MDQxNDA2MTY1M1oXDTI3MDQxMjA2
    MTY1M1owRDEcMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEkMCIGA1UEAwwb
    cm9vdEB0aW1zZXJ2ZXIudGltc3N0dWZmLmV1MIICIjANBgkqhkiG9w0BAQEFAAOC
    Ag8AMIICCgKCAgEAw5dwGlKwUQNFQlORZYK/SApR+xEEiu0rxM62GysFztuF41ps
    g2ig7MyTZ9yTRFm4WngkVVkamXswkO/l7nQ9VNYfIE/HFR76hbo/QcLbx6SM25U7
    DigzlsPj7UpN2BwTHdNj8y55JnXfqNnbN5Qvb2hP3IJg2e6TfgpmLNxoeoOid9Ju
    0wLGT6OaVvIjQv2AV51hDjORjoVB2yELgGu81DjPnVloKmmaDqIfDYAxj51EjIeR
    J+ELZ53zwYaDuMAdDFFykyJ0EJCjOzfBMrWfsVUmhUiwgUSZ/3u9fFq9N6l0mVws
    pBd/jCNROOOQB0GEZU6j7H/mab3ZHmyrSC9KmJAiFhtYwWLGcBZmWQWmEV39rC+E
    w598TKSRzsz4vhzrBwwGVxHA6IixKQU86Nncou90NIhIB6J1QD7bLiAwk4gxOgJV
    V4ouPM7zmiDn3EWe/vGuMc1ugciWTtotJaQ54o0+DLqWW3ZIdRYTTov/vMuYrI3C
    WMNGbT51idjV8M4UoHo2eMGSIsay9m9p+aYORfdEHSg+8+2i9+D4fjb0sMeSlEpi
    YXpljdyAxh9w/hIDbIoKbhvo8jqxGbUsG3mHvE2TWXfmTkWOBrbcWA5uePNscHOV
    HW7htUpniEWo+7JQcsiPgpRepf7IHI7c2Kvc1lBOxpPpBdRq/duFpwuW5CECAwEA
    AaNeMFwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1Ud
    EwEB/wQCMAAwJwYDVR0RBCAwHoIWdGltc2VydmVyLnRpbXNzdHVmZi5ldYcEwKgA
    BDANBgkqhkiG9w0BAQsFAAOCAgEAcP0pir9ZIdgG9SWk6aoWSrl/8d9TZhxzO4Hm
    i0SshsV6NeZJbrRwkmNjyaBQKTFMgPBGAaUGuKdfCg71ixol84u+yUxjeJfF8ISc
    MkP9ObEAGQXVwBHz7S6DqsGXvLDNWfjI+NSquIMV1i0CE7pD0Nyp8WO+AO7QdUJW
    jqD19yOZ1Jho7dOuCTItWFFXLbP+M+zWhDislw4v8U/MjpMiRSoYpkCycLpaHiEv
    6hcSHNcZaUPh+0wDJFt6sbX1xV6pTjw9jta50CpVKrYncn9qt1jLEqgOI4GI5HUr
    0fRB0ADXmcGnLxyk5zVg7Mf3ZUxPiJv4Xf+T9BuCKM6SrBa91piijIUYKpkNQ8zL
    tNk7ciFKMqq6/R80I2wuLueVHOARCsIfEtnz4VMbuWYd93wayosK2zGwuP3UP+RS
    NTfCCKNtf9tkawZB9rUFMu1Wdj/EccOSqteVMqwXaaGEGjJ6STcEhoUuDKemljBX
    Bsdx8C8U71vUTrilGpdzsTlwn+EPXK3g7YgfQZqQRzF80k5iCvOK9H3fuR49xsy9
    VOkX+w51+hSyBe4sZsiUKg4zgi5OfBLJUjt2e9L16d1TOO3bd5tjr4U7TijBqXyL
    jBI8pe+9mcYAz1fJ91O/AWjewI7xPPQdUDJREOjxg+jqYLCYLla/79n6kHiLiV90
    NhqIjLU=
    -----END CERTIFICATE-----
  certificate_fingerprint: 3b48c7a09949167bcbd30f23ccd1511f19fcde2f15e64efc4e636f158f831282
  driver: lxc
  driver_version: 2.0.8
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.4.0-78-generic
  server: lxd
  server_pid: 26238
  server_version: "2.13"
  storage: btrfs
  storage_version: "4.4"

Issue description
lxc file push --recursive doesn't support symlinks
Steps to reproduce

cd /tmp/
mkdir testdir
touch testdir/normalfile
ln -s normalfile testdir/linkto_normalfile
$ ls -l testdir/
total 0
lrwxrwxrwx 1 user user 10 May 21 19:28 linkto_normalfile -> normalfile
-rw-rw---- 1 user user  0 May 21 19:27 normalfile

$ lxc file push --recursive testdir/ local:container/tmp/
error: 'testdir/linkto_normalfile' isn't a regular file or directory.

Workaround is, for example to copy /home/username/Music over:
tar cf - -C /home/username Music | lxc exec local:"$1" -- tar xvf - -C /home/username
So many little gotchas with LXC/LXD still, it really is painful to setup I hate to say. Although very useful once running

To make this possible, we'd need to support the creation of symlinks in the file API. Right now the only types support are directory or file, symlink would be its own separate type with the content being the target.

I was thinking more that symlinks should be created inside the container, rather than the contents copied over. It'd be a bit of a change in behaviour to what most people expect - e.g. tools like cp or rsync don't copy the contents of symlinks but instead copy the symlink to the target. If the symlink's not valid any more on the target that's up to the user to decide to ignore or fix, not cp or rsync.
In my particular case it was a directory of configuration data and scripts which I wanted to run in the container, this directory had some relative links that would still have been valid anyway inside the container.

(well technically cp and rsync can copy symlinks as the contents, but you have to explicitly specify that with -L or whatever the option is, it's not the default)

That's what I meant. When I said "the content being the target" I mean, the content of the HTTP request would be the target path as a string.

curl --unix-socket /var/lib/lxd/unix.socket -s lxd/1.0/containers/blah/files?path=/tmp/a -H "X-LXD-type: symlink" -H "Content-Type: text/plain" -d "/tmp/b"

That would create a symlink at /tmp/a which points to /tmp/b

ok, understood. I didn't realise the lxc file stuff was implemented as HTTP requests, I assumed it was file system operations, but makes sense

All communications between the client tool and LXD are done as HTTP requests using our REST API.

stgraber@castiana:/tmp$ mkdir testdir
stgraber@castiana:/tmp$ touch testdir/normalfile
stgraber@castiana:/tmp$ ln -s normalfile testdir/linkto_normalfile
stgraber@castiana:/tmp$ ls -l testdir/
total 0
lrwxrwxrwx 1 stgraber domain admins 10 Jul  2 01:02 linkto_normalfile -> normalfile
-rw-r--r-- 1 stgraber domain admins  0 Jul  2 01:01 normalfile
stgraber@castiana:/tmp$ lxc file push -r testdir/ c1/tmp/
stgraber@castiana:/tmp$ lxc exec c1 -- ls -lh /tmp/
total 1
drwxr-xr-x    2 201105   200512         4 Jul  2 05:02 testdir
stgraber@castiana:/tmp$ lxc exec c1 -- ls -lh /tmp/testdir/
total 1
lrwxrwxrwx    1 201105   200512        10 Jul  2 05:02 linkto_normalfile -> normalfile
-rw-r--r--    1 201105   200512         0 Jul  2 05:02 normalfile
stgraber@castiana:/tmp$

Thanks! I've now had time to test this and it works as far as copying symlinks goes.
Unfortunately I hit another bug relating to files with spaces in their names, see https://github.com/lxc/lxd/issues/3487


Funder and sample plan links don't save properly

@briri added fields for Funder and Sample plan links and link text to the template details editing page, but they aren't saving properly. When I enter a link and link text and click save I receive the expected confirmation message (screenshot 1). But when I open the template details editing section accordion, the URL is copied in the link text field (screenshot 2). This also happens w/Sample plans link fields.
Desired behavior is to preserve the link and link text as entered. Also when I click Save, the template details accordion should remain open/expanded so I can verify that the new info is saved. Current behavior closes the accordion.

@stephaniesimms, I'm assuming this issue is raised against a code that includes among other things, the PR #902. The current code that CDL-MVP branch has, does allow you to save either funder links or sample plan links.

@jollopre this is occurring in the CDL-MVP branch that is currently deployed on roadmap-stg

@briri, if by not-saving you mean leaving empty either URL or link text of any new record created through the interface, YES, it does not save since there is an if statement preventing that in the JS. Can you provide an scenario that you consider it is failing. Thanks

Log on to roadmap-stg, change your org affiliation to a funder org, go to a template details page, enter a link (e.g. link: http://google.com, text: Google), click save
The page refreshes and shows link: http://google.com and text: http://google.com

It’s the issue reported in the main text of the ticket above

@briri for the explanation. I tweaked that, it was not a persisting problem, like the issue title suggests, that's the reason I was confused.

@briri is addressing related issues w/saving links in #866 and #904 - i'll just leave this here to await testing all together.

i can enter and save links now. but there doesn't appear to be a validation step to ensure I entered a valid URL and link text. this is also the case for links in the Org details page. we need the validation step in both places because the links are displayed on a public page and in the org branding banner. another missing piece is that the Funder and Sample plan links entered here should appear in the DMP Templates page. i can create a new issue(s) as needed @jollopre @briri
One thing that can be addressed in this one:

Desired behavior is to keep the template details accordion open/expanded when I click Save so I can verify that the new info is saved. Current behavior closes the accordion.

The URL validation will be a part of the changes for #904 which is currently in the Review column

@stephaniesimms, PR #966 allows the user to keep in the template details when edit.


Please allow up to four parameters for --titlefont, --messagefont

We have a corporate standard font installed on every Mac, and a standard corporate colo(u)r palette.  I would like to use titlefont to set the color, size, font family AND weight all at once.  With only three options, I will need to compromise on size, weight, or color.
Please allow all four parameters to be used.  Thanks!

Unintended workaround, possible bug:
The 'weight' option doesn't appear to do anything here (or yet).
However, the "name" field accepts both the "PostScript Name" and "Full Name" of a given font / weight.
This results in SF Pro Regular
dialog -d -t "ACME Corp Software Updates" --titlefont size=24,name=SFPro,weight=heavy
dialog -d -t "ACME Corp Software Updates" --titlefont size=24,name=SFPro,weight="heavy"
dialog -d -t "ACME Corp Software Updates" --titlefont size=24,name="SF Pro",weight=light
Results in SF Pro Heavy:
dialog -d -t "ACME Corp Software Updates" --titlefont size=24,name=SFPro-Heavy
dialog -d -t "ACME Corp Software Updates" --titlefont size=24,name="SF Pro Heavy"
Results in SF Pro Light:
dialog -d -t "ACME Corp Software Updates" --titlefont size=24,name="SF Pro Light"

Due to how the title font stuff is written, weight only takes effect when not specifying the font. To change the weight of specified font you need to use the full font name, as you discovered SFPro-Heavy or the like.
--listfonts will list out all the names that will be recognised. This may match the names in font book or it might be different.
There are optimisations that can be made though.


[Keyboard Navigation - Power Toys - Power Rename]: 'Original/Renamed' header controls present in 'Power Rename' dialog are not accessible using keyboard.

[Power Toys Utility - Power Rename]
User Experience:
Keyboard only users will find it very difficult to access a control if it doesn't get activated using keyboard and if they are accessible using mouse.
Test Environment:
OS Version: 20221.1000
App Name: Power Toy Preview
App Version: v0.23.0
Screen Reader: Narrator
Repro Steps:

Open Power Toys Settings App.
Open File explorer and select multiple files and open context menu and select Power rename. 'Power Rename' dialog will open.
Navigate to 'Original/Renamed' header controls present in 'Preview' table using keyboard.
Observe the issue.

Actual Result:
'Original/Renamed' header controls present in 'Power Rename' dialog are not accessible using keyboard. It is accessible using mouse only.
Note:
This issue is repro throughout the app.
Expected Result:
'Original/Renamed' header controls present in 'Power Rename' dialog should be  accessible using both keyboard and mouse.
MAS Reference:
https://microsoft.sharepoint.com/:w:/r/teams/msenable/_layouts/15/WopiFrame.aspx?sourcedoc={8492c4eb-c179-40ae-8777-cd044ed725a2}&cid=7bc42b51-9ddd-4d01-89e8-1081fff45d03
37_MAS2.1.1_Original_rename_eader_control_not_accessible_keyboard.zip
37_UIA.zip

This may be a bug in the ComCtl32 library based on this comment https://github.com/dotnet/winforms/issues/1726#issuecomment-677988114

In accordance with internal accessibility requirements, we are closing this issue as obsolete in favor of #12869 as we plan the relevant UI updates for PowerRename.


build from source ends with error

Following instructions on building from source resulted in this error:
:~/wallet713/wallet713$ cargo build --release
    Updating registry `https://github.com/rust-lang/crates.io-index`
    Updating git repository `https://github.com/mimblewimble/grin`
error: no matching package named `grin_api` found
location searched: https://github.com/mimblewimble/grin#c41ea4bc
required by package `wallet713 v1.0.1 (file:///home/urza/wallet713/wallet713)`

Hey @urza, not sure what this is about :) There's definitely a grin_api package in the grin repo (https://github.com/mimblewimble/grin/tree/master/api). Did it eventually work for you?

Hi @ravidio , I didnt make any more attempts.. are there perhaps some rust commands that I may try? maybe something wrong in my rust/cargo cache? I was building grin a while back on this machine (that I since deleted and use grin binary releases). Sorry I am not at all familiar with rust ecosystem and how to properly use cargo :) just copy pasting commands from grin and wallet713 readmes :)

Can we assume this is fixed? Latest master should build without problem. If you do still have issues, please re-open.


How to reduce CPU occupancy？

The jar package started on Linux has a very high CPU share when calling this HtmlUnit utility class. Is there any way to reduce it?
:(

Please provide more details - e.g. what utility you call

Sorry.
I wrote a recursive method to access more pages using htmlunit to get more business content.
When there is a request, the CPU share is suddenly very high, and then it gradually stabilizes.
Are there any methods and suggestions for improvement?
Thank Q!
         public void doPagesReptileBFs(String nowHtml, int deep) {
		if (deep > 10)return;
		urlMap.put(nowHtml, "right");
		List<String> htmlList = new ArrayList<>();
		HtmlPage page = null;
		try {
			page = webClient.getPage(nowHtml);
		} catch (IOException e) {
			e.printStackTrace();
		}
		if (page == null || page.getWebResponse().getStatusCode() != 200)return;

		DomNodeList<DomElement> domList = page.getElementsByTagName("a");
		if (domList == null || domList.getLength() == 0)return;
		for (DomElement dom : domList) {
			// URL processing...
			if(urlMap.get(URL) != null)continue;
			doPagesReptileBFs(URL, deep+1);
		}
	}

First comment:
waitForBackgroundJavascript is not an setting, this call really waits - because you have not opened an url the call is useless.

Not sure what kind of cpu you are using but you program  runs without any pause. Because there is a main thread and at least one javascript thread i think two cpu's are more or less complete busy.
Depending on the pages you might consider

disabling javascript support
intercept the analytics stuff and remove it

But in general HtmlUnit tires to process your program as fast as possible - this implies to us the available cpu resources.

Ok, thank you!!! : )

Your welcome, will close this. Feel free to open a new one if you need more


[BACKLOG-8248] Removed connect and disconnect from repository menu

Base build failed with return code 1

@mdamour1976


Tags with < and > fail to parse

I setup rabbitmq with autoscaling module in AWS Autoscaling environment. They did not start, with errors in logs:
=CRASH REPORT==== 26-Jan-2017::08:43:08 ===
  crasher:
    initial call: application_master:init/4
    pid: <0.140.0>
    registered_name: []
    exception exit: {bad_return,
                        {{rabbit,start,[normal,[]]},
                         {'EXIT',
                             {{function_clause,
                                  [{rabbitmq_aws_xml,flatten_text,
                                       [[">|, u|groupSet|: {u|items|: [{u|groupId|: u|sg-a8f540d4|}]}, u|instancesSet|: {u|items|: [{u|minCount|: 1, u|maxCount|: 1, u|imageId|: u
|ami-df866cc9|}]}, u|clientToken|: u|5f001b98-3743-4cd4-83cd-f917df35e59e_subn",
                                         "<sensitiveDataRemoved",
                                         "u|userData|: u|"],
                                        []],
.....
                              {gen_server,call,
                                  [rabbitmq_aws,
                                   {request,"ec2",get,[],
                                       "/?Action=DescribeInstances&InstanceId.3=i-002524e813d6e8840&Version=2015-10-01",
                                       [],[],undefined}]}}}}}
      in function  application_master:init/4 (application_master.erl, line 134)

We have special tags for security management of our servers. This tags could contain special symbols, like '>', '<', etc. I removed this tags and rabbit had started properly.
It looks like bug in module and it don't escape special characters

@opokhvalit can you please provide an example EC2 response with such tags? (this plugin lists instances)
It would be great to have more specific examples than "something with an angle bracket".

you can find the simple example in the log above. But anyway I will reproduce this trouble and put here more detais

How to reproduce the problem.
Put tag on instance with any name and value equal to (this is part of data from an amazon event):
u|userData|: u|<sensitiveDataRemoved>|, u|groupSet|: {u|items|: [{u|groupId|: u|sg-aaaaaaaa|}]}, u|instancesSet|: {u|items|: [{u|minCount|: 1, u|maxCount|: 1, u|imageId|: u|ami-df866cc9|}]}, u|clientToken|: u|802c996d-2a16-45f3-93e7-72e301dd7fe3_subn
after that try to start rabbitmq with config like:
[
    {rabbit, [        {log_levels, [{connection, warning},{autocluster, info}]}
    ]},
    {autocluster, [
        {backend, aws},
        {aws_autoscaling, true},
        {aws_ec2_region, "us-east-1"}
    ]},
    {rabbitmq_stomp, [{tcp_listeners, [{"<IP_ADDRESS>", 6163}]}]}
].

RabbitMQ version is 3.6.6, OS - Linux, erlang version 19.2.0

This plugin was forked by the RabbitMQ team and is now part of RabbitMQ. More information can be found @ https://github.com/rabbitmq/rabbitmq-autocluster


doc: Clean unresolved git conflicts

Summary

Cleans up a few unresolved git conflicts that slipped through from #42
Checklist

[ ] Scope: The referenced issue is addressed, there are no unrelated changes.
[X] Run Locally: Verified that the docs build using make server
[ ] New Features Marked: Documentation for new features is marked using the new-in-version shortcode, according to the guidelines in CONTRIBUTING.
[X] Style Guidelines: Documentation obeys style guidelines in CONTRIBUTING.
[X] Commits: Commit messages follow guidelines in CONTRIBUTING, there are no fixup commits left.

Closed, was already part of #81


No config.json

Hey guys,
I seem to be missing the config file and actually the homebridge dir as well
Homebridge works ok, I can find the bridge in the home app, I have installed the LIFX plugin. But thats as far as I can get?
Any ideas, im pretty new to Linux, but it all seemed to install ok?

Sorry, figured it out :)

PLEASE answer your own question. Why would you just answer with "Sorry, figured it out?". How will that help the next guy that has the same problem as you?
I do understand that the answer to this question is very trivial, but come on, don't be that guy that asks questions, but never provide the solution.

HAHA, you're right. I was rushing though it last night, I was going to update :)
For those of you that cant find the config-sample.json, following the guides for the Ubuntu install, you will find it @ /usr/lib/node_modules/homebridge
Also WIndows people using Linux, use nano as the text editor, less googling that way, much more intuitive.


Typos in documentation

Hi @carlosuc3m,
There seems to be some typos in the quickstart instructions in the ReadMe - these modifications worked for me and allowed it to run.

Hello again @djpbarry I just wanted to ask you where did you get the error in the Jython scripts? If you are using a M1, some of the engines are not compatible with Rosetta.
REgards,
Carlos

Okk now I understand. You were probaly trying to load Tf 2.10 in Fiji, however it requires JAva 11 and Fiji only supports JAva 8. The comptiblity between JAva versions and engines is specified in the JSOn enignes file but I will make it more visible.


Label rotation for x axis

When there are enough data bars, some names are hidden even though its shown in the legend. A solution could be to allow to rotate the x labels by some degree to allow more names to be shown or even make it automatically given the amount of bars and space available

Hey @angelhodar, thank you for using our library. This sounds reasonable, thank you for reaching out. We will consider this feature for one of our next releases.

Recharts reference: https://recharts.org/en-US/api/XAxis#angle

Hello all, after considering this for a long time, we decided to not implement this feature. In general, we would have never used this ourselves, and rather opted to create shorter labels or even better, use the layout = "vertical" option. Thank you all for sticking around and being part of this community!


[Feature Request] Add support for WARC file format

WARC is well-known format for storing crawled captures. It can store arbitrary number of HTTP requests and responses along with other network interactions such as DNS lookups along with their header, payload, and other metadata. It is usually used by web archives, but there are some other use cases as well. WARC is the default format in which Heritrix crawler (originally developed by the Internet Archive) stores captures. Wget supports WARC format as well. There are some other tools such as WARCreate (a Chrome extension) to save web pages in WARC format along with all their page requisites while browsing and Squidwarc (a Headless Chrome-based crawler) specifically for archival purposes.
That said, adding support for WARC format will immediately make this project more useful for the web archiving community.

Thanks for filing an issue, but I don't think we should invest into this in near future.
I understand web archive is of solid need, but I also believe libraries should be dedicated to that specific purpose like Heritrix crawler and Squidwarc you mentioned.
I'd like to focus this crawler mainly on scraping purpose. My assumption is that people who want to take web archiving do not need to scrape HTML elements at the same time, and vice versa.
I'll keep this issue opened for a while to hear more opinions and use cases from others.

I'm interested in web archiving too and played a bit with the crawler and since it exposes puppeteer's Page it's possible to intercept the traffic and save it as WARC.
I was trying to do it with node-warc, tampering a little bit with the library code to expose the WARCWriterBase class.
const HCCrawler = require('headless-chrome-crawler')
const WARC = require('node-warc')

const run = async () => {
	const crawler = await HCCrawler.launch({
		args: ['--disable-web-security', '--ignore-certificate-errors', '--allow-running-insecure-content'],
	})

	crawler.on('newpage', async page => {
		await page.setRequestInterception(true)

		page.on('request', async request => {
			request.continue()
		})

		page.on('response', async response => {

			const assetWriter = new WARC.WARCWriterBase()
			assetWriter.initWARC('./temp_2.warc', true)

			let reqHeaders = ''
			for (header of Object.entries(response.request().headers())) {
				reqHeaders = `${reqHeaders}${header.join(': ')}\n`
			}

			await assetWriter.writeRequestRecord(response.request().url(), reqHeaders, response.request().postData())

			const responseData = await response.buffer()

			let resHeaders = ''
			for (header of Object.entries(response.headers())) {
				resHeaders = `${resHeaders}${header.join(': ')}\n`
			}

			await assetWriter.writeResponseRecord(response.url(), resHeaders, responseData)

		})
	})

	crawler.queue('https://google.com/')

	crawler.onIdle().then(() => crawler.close())
}

run()

But the WARC that come's out isn't well formated and I'm not sure why. I'm not an expert in WARC files and crawling, and couldn't come up with a reason which it doesn't work. The library has a great API but I've found a few things that don't make sense acording to the WARC Specifications
It would be great to continue exploring this and also make a PR to close https://github.com/N0taN3rd/node-warc/issues/2

Thanks @BubuAnabelas for your investigation. I will ping @N0taN3rd here to see why node-warc is misbehaving. However, if you have a more concrete description of the issue, you might want to create a ticket in its repository.

I'm sorry I don't have a more concrete description of the issue, nevertheless, there's the script I used which (in theory) would give you the same WARC file as mine. With that file you could try to parse it with some other tool or text editor and view the different problems you can identify.
For instance there are some records in which it seams that the request/response headers and the request data/response body are not written. It might be an asynchronous problem but that's why those await are there and it stills happens.
I also tried disabling the page cache by adding await page.setCacheEnabled(false) to the code but nothing changed.
Perhaps @N0taN3rd can help us and I'll contribute opening the necessary issues in node-warc's repo


New test nnbd failures on dartkp-strong-linux-release-x64

There are new test failures on [vm] Reify loading units....Add WorkspacePackage to each FileState..
The tests
co19/LanguageFeatures/nnbd/type-normalization/equations_A12_t01 Crash (expected Pass)
co19/LanguageFeatures/nnbd/type-normalization/norm_A02_t04 Crash (expected Pass)
co19/LanguageFeatures/nnbd/type-normalization/norm_A02_t05 Crash (expected Pass)
co19/LanguageFeatures/nnbd/type-normalization/norm_A02_t07 Crash (expected Pass)
co19/LanguageFeatures/nnbd/type-normalization/norm_A06_t10 Crash (expected Pass)

are failing on configurations
dartkp-strong-linux-release-x64

/cc @rmacnak-google @alexmarkov

Failure log:
/======================================================================================================================================\
| dartkp-strong-linux-release-x64:co19/LanguageFeatures/nnbd/type-normalization/equations_A12_t01 broke (Pass -> Crash, expected Pass) |
\======================================================================================================================================/

--- Command "vm_compile_to_kernel []" (took 02.000862s):
DART_CONFIGURATION=ReleaseX64 /b/s/w/ir/pkg/vm/tool/gen_kernel --aot --platform=out/ReleaseX64/vm_platform_strong.dill -o /b/s/w/ir/out/ReleaseX64/generated_compilations/dartkp/tests_co19_src_LanguageFeatures_nnbd_type-normalization_equations_A12_t01/out.dill /b/s/w/ir/tests/co19/src/LanguageFeatures/nnbd/type-normalization/equations_A12_t01.dart --enable-experiment=non-nullable --packages=/b/s/w/ir/.packages -Ddart.vm.product=false -Ddart.developer.causal_async_stacks=true --null-safety

exit code:
0

--- Command "precompiler" (took 94ms):
DART_CONFIGURATION=ReleaseX64 out/ReleaseX64/gen_snapshot --enable-experiment=non-nullable --snapshot-kind=app-aot-assembly --assembly=/b/s/w/ir/out/ReleaseX64/generated_compilations/dartkp/tests_co19_src_LanguageFeatures_nnbd_type-normalization_equations_A12_t01/out.S --null-safety --enable-experiment=non-nullable --ignore-unrecognized-flags --packages=/b/s/w/ir/.packages /b/s/w/ir/out/ReleaseX64/generated_compilations/dartkp/tests_co19_src_LanguageFeatures_nnbd_type-normalization_equations_A12_t01/out.dill

exit code:
-6

stderr:
===== CRASH =====
si_signo=Segmentation fault(11), si_code=1, si_addr=0xaf
version=2.9.0-edge.d4ffb92ed8fb33d4a7597a513b4e5556d4439ed7 (be) (Thu Jul 9 00:15:07 2020 +0000) on "linux_x64"
pid=5723, thread=5723, isolate_group=isolate(0x564461a0e500), isolate=isolate(0x5644619fdb00)
isolate_instructions=0, vm_instructions=0
  pc 0x000056445f19d497 fp 0x00007ffd94aa1ce0 dart::Type::IsInstantiated(dart::Genericity, long, dart::ZoneGrowableHandlePtrArray<dart::AbstractType const>*) const+0x127
  pc 0x000056445f19d29b fp 0x00007ffd94aa1d00 dart::Type::SetIsFinalized() const+0x1b
  pc 0x000056445f19b98b fp 0x00007ffd94aa1d40 dart::AbstractType::NormalizeFutureOrType(dart::Heap::Space) const+0x17b
  pc 0x000056445f655bc3 fp 0x00007ffd94aa1d80 dart::kernel::TypeTranslator::BuildInterfaceType(bool)+0x323
  pc 0x000056445f656b68 fp 0x00007ffd94aa1db0 dart::kernel::TypeTranslator::BuildTypeArguments(long)+0x98
  pc 0x000056445f655b38 fp 0x00007ffd94aa1df0 dart::kernel::TypeTranslator::BuildInterfaceType(bool)+0x298
  pc 0x000056445f65587d fp 0x00007ffd94aa1e10 dart::kernel::TypeTranslator::BuildTypeWithoutFinalization()+0x1d
  pc 0x000056445f13f9b1 fp 0x00007ffd94aa1e60 dart::kernel::KernelLoader::LoadPreliminaryClass(dart::kernel::ClassHelper*, long)+0x71
  pc 0x000056445f13ddf1 fp 0x00007ffd94aa1f90 dart::kernel::KernelLoader::LoadClass(dart::Library const&, dart::Class const&, long, dart::Class*)+0x2e1
  pc 0x000056445f13c635 fp 0x00007ffd94aa20b0 dart::kernel::KernelLoader::LoadLibrary(long)+0x555
  pc 0x000056445f13a97b fp 0x00007ffd94aa21e0 dart::kernel::KernelLoader::LoadProgram(bool)+0x14b
  pc 0x000056445f06b2c6 fp 0x00007ffd94aa26d0 out/ReleaseX64/gen_snapshot+0x5a02c6
  pc 0x000056445f06ae31 fp 0x00007ffd94aa2760 dart::Bootstrap::DoBootstrapping(unsigned char const*, long)+0x281
  pc 0x000056445f15c620 fp 0x00007ffd94aa2870 dart::Object::Init(dart::Isolate*, unsigned char const*, long)+0xc010
  pc 0x000056445f0d10f6 fp 0x00007ffd94aa2930 dart::Dart::InitIsolateFromSnapshot(dart::Thread*, dart::Isolate*, unsigned char const*, unsigned char const*, unsigned char const*, long)+0x46
  pc 0x000056445f0d15a6 fp 0x00007ffd94aa2a10 dart::Dart::InitializeIsolate(unsigned char const*, unsigned char const*, unsigned char const*, long, dart::IsolateGroup*, void*)+0xc6
  pc 0x000056445f68f7dd fp 0x00007ffd94aa2ab0 out/ReleaseX64/gen_snapshot+0xbc47dd
  pc 0x000056445f690143 fp 0x00007ffd94aa2b90 Dart_CreateIsolateGroupFromKernel+0x1c3
  pc 0x000056445f01e98c fp 0x00007ffd94aa2d20 dart::bin::main(int, char**)+0x70c
-- End of DumpStackTrace

--- Re-run this test:
python tools/test.py -n dartkp-strong-linux-release-x64 co19/LanguageFeatures/nnbd/type-normalization/equations_A12_t01

/cc @crelier

Class Future is needed for type normalization but is not yet cached in the object store.
CL https://dart-review.googlesource.com/c/sdk/+/153901 resolves the class on the fly when not yet cached.

Fixed by https://dart-review.googlesource.com/c/sdk/+/153901


Remove unnecessary copy when iterating keys of a Config

archaius-pull-requests #275 SUCCESS
This pull request looks good

NetflixOSS » archaius » archaius-pull-requests #106 SUCCESS
This pull request looks good

NetflixOSS » archaius » archaius-pull-requests #107 FAILURE
Looks like there's a problem with this pull request

archaius-pull-requests #276 FAILURE
Looks like there's a problem with this pull request

archaius-pull-requests #277 SUCCESS
This pull request looks good

NetflixOSS » archaius » archaius-pull-requests #108 SUCCESS
This pull request looks good


Replace os.plateform()  and os.arch()

Replace os methods by the cached process properties

Why is this needed?


A tour of Go @ Appending to a slice

What version of Go are you using (go version)?
go1.9.3
Does this issue reproduce with the latest release?
I think I'm using the latest release
What operating system and processor architecture are you using (go env)?
darwin/amd64
What did you do?
var s []int
printSlice(s)

// append works on nil slices.
s = append(s, 0)
printSlice(s)

// The slice grows as needed.
s = append(s, 1)
printSlice(s)

// We can add more than one element at a time.
s = append(s, 2,3,4)
printSlice(s)

What did you expect to see?
len=0 cap=0 []
len=1 cap=1 [0]
len=2 cap=2 [0 1]
len=5 cap=5 [0 1 2 3 4]
What did you see instead?
len=0 cap=0 []
len=1 cap=1 [0]
len=2 cap=2 [0 1]
len=5 cap=6 [0 1 2 3 4]

Hi,
the capacity of slice gets doubled in general (for not so large length array) whenever the slice outgrows its current capacity.
So expected capacity should have been 8 instead of 6 (as printed by code) or 5 (as expected by you).
When 2 is added to the slice, it exceeds its current capacity and new slice with double the capacity is created. So the capacity becomes 4. Next when 3 is added, capacity is not exceeded;hence it remains unchanged. But finally when 4 is appended to slice , its capacity increases the current capacity of 4 and new slice with capacity 8 (double the current capacity of 4) is created.
please refer following blog for more information on slices :
https://blog.golang.org/go-slices-usage-and-internals
BR.

As explained above, this is working as intended. Closing here.

Thanks!

@WheeskyJack
Hi WheeskyJack, I was still confused about s = append(s, 0), after that why cap turns to 2?
is it (cap+1)*2 when cap is 0?


use class instead of interface for NatsError declaration

This allows users to use NatsError with instanceof:
if (e instanceof NatsError) {
    /** handle */
}

The generation of that interface is actually the typescript compiler. Not sure why it does that, as the types it exactly like that:
https://github.com/nats-io/nats.deno/blob/main/nats-base-client/error.ts#L80

Alright, so I'll close this PR

@cuddlecake the question is whether we want to tweak the generated output.... what do you think?

@aricart so, the change I've made with NatsError would be very useful for differentiating errors, especially if the functions isAuthError etc. would be available as well.
Another thing that I noticed is that the extension with AsyncIterable at Subscription: export interface Subscription extends AsyncIterable<Msg> does not work in TypeScript (it does not recognize that it is an AsyncIterable and I have to put @ts-ignore on the usage. On the other hand, adding the field [Symbol.asyncIterator](): AsyncIterator<Msg> in the Subscription, the compiler doesn't complain. This one is especially weird, since the AsyncIterator interface should be doing the same. It might just be an issue with my tsconfig.

I don't show that error on my side, and the typescript code as written is compiled by tsc on the node.js side. https://github.com/nats-io/nats.js/blob/master/tsconfig.json


Feature: Load model from BMZ using URL

Description

What: Now it is possible to pass a URL to the load_from_bmz function to download and load BMZ files.
Why: Not many users will have access to the model resource URLs, but this functionality is useful for developing the CAREamics BMZ compatibility script.
How:

Type hint path as also pydantic.HttpUrl in load_from_bmz (as in bioimage.core);
Remove path validation checks from load_from_bmz and allow it to be handled in load_model_description
Call download on the file resources to download and get the correct path.

Changes Made

Modified:

load_from_bmz
extract_model_path

Additional Notes and Examples
This will have merge conflicts with #271.
There are currently no official tests (it does work), we can discuss using the URL of one of the existing CAREamics models uploaded to the BMZ or create a Mock.

Please ensure your PR meets the following requirements:

[x] Code builds and passes tests locally, including doctests
[ ] New tests have been added (for bug fixes/features)
[x] Pre-commit passes
[ ] PR to the documentation exists (for bug fixes / features)

A test with one of our model would be a good idea indeed!
Hopefully it doesn't throw off the CI too often due to bandwidth or unavailable BMZ.

@jdeschamps For now I will not add a test because there are currently no suitable models to test on, and I am using existing functionality available in bioimageio.core and bioimageio.spec(that I hope is already tested 😄). Any API changes with thebioimageio` should be caught by our existing tests.
Still, it would be nice to have an integration test that downloads our models so I will open an issue as a reminder to make one once the models have the correctly named config file.


Stamp Labels are missing in the "Add postage stamp" page of update feeds

This is in PostageStampSelector.
If a label is set on a stamp, it should show the label, and then the short hash in parantheses.


[Bug]: LocalEntryNotFoundError encountered while instantiating OpenCLIPEmbeddingFunction

What happened?
I am a newbie and want to try chroma's multimodal embedding, so I followed the code given in chroma's official website, but I encountered the following problem:

Versions
chroma-hnswlib                           0.7.6
chromadb                                    0.5.5
(I just ran pip install chroma on 2024.8.20)
python                                         3.11.9
Ubuntu                                        22.04
Relevant log output
from chromadb.utils.embedding_functions import OpenCLIPEmbeddingFunction
embedding_function = OpenCLIPEmbeddingFunction()

---------------------------------------------------------------------------
gaierror                                  Traceback (most recent call last)
File ~/miniconda3/envs/chroma_test2/lib/python3.11/site-packages/urllib3/connection.py:196, in HTTPConnection._new_conn(self)
    195 try:
--> 196     sock = connection.create_connection(
    197         (self._dns_host, self.port),
    198         self.timeout,
    199         source_address=self.source_address,
    200         socket_options=self.socket_options,
    201     )
    202 except socket.gaierror as e:

File ~/miniconda3/envs/chroma_test2/lib/python3.11/site-packages/urllib3/util/connection.py:60, in create_connection(address, timeout, source_address, socket_options)
     58     raise LocationParseError(f"'{host}', label empty or too long") from None
---> 60 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
     61     af, socktype, proto, canonname, sa = res

File ~/miniconda3/envs/chroma_test2/lib/python3.11/socket.py:962, in getaddrinfo(host, port, family, type, proto, flags)
    961 addrlist = []
--> 962 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    963     af, socktype, proto, canonname, sa = res

gaierror: [Errno -3] Temporary failure in name resolution

The above exception was the direct cause of the following exception:
...
   1859         " in the local cache. Please check your connection and try again or make sure your Internet connection"
   1860         " is on."
   1861     ) from head_call_error

LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

I changed the code a bit according to the github repository, but it still reports the same error:

hey @TaugenichtsZZY, it seems like OpenCLIP is trying to download thet or the tokenizer config and fails due to a model checkpoint lack of connectivity to the internet or an intermittent error. The second image you posted suggests a DNS resolution failure.
Are you trying to run this a machine without internet connection?

hey @TaugenichtsZZY, it seems like OpenCLIP is trying to download thet or the tokenizer config and fails due to a model checkpoint lack of connectivity to the internet or an intermittent error. The second image you posted suggests a DNS resolution failure.

Thanks for the reply. My runtime environment is in vscode, I have internet connection, and I can ping google.com normally from the command line. But this error always happens whether I use vpn or not. Is there a solution for this please?

hey @TaugenichtsZZY, it seems like OpenCLIP is trying to download thet or the tokenizer config and fails due to a model checkpoint lack of connectivity to the internet or an intermittent error. The second image you posted suggests a DNS resolution failure.

I've tried changing dns <IP_ADDRESS> and <IP_ADDRESS> and it doesn't seem to help much. Or is there another way to use the multimodal embedding model in chroma? I really want to use chroma for my research!

Hi @TaugenichtsZZY, using the OpenCLIP embedding function with Chroma requires that you're able to download the required model from the internet - it seems that you might be using an ISP or else have some other network issue that's preventing you from doing so.
Were you able to install the open_clip module, via pip install open-clip-torch ? Are you able to run the example on the OpenCLIP repo? https://github.com/mlfoundations/open_clip ?

Hi @TaugenichtsZZY, using the OpenCLIP embedding function with Chroma requires that you're able to download the required model from the internet - it seems that you might be using an ISP or else have some other network issue that's preventing you from doing so.
Were you able to install the open_clip module, via pip install open-clip-torch ? Are you able to run the example on the OpenCLIP repo? https://github.com/mlfoundations/open_clip ?

Thanks for the reply. I'm in mainland China. If by ISP you mean Internet Service Provider, it is indeed possible that this is the problem. But the strange thing is that I have already set up a proxy on my computer using Clash.
I actually executed pip install open-clip-torch from the command line and successfully installed the package, but I think it should work in vscode as well.

You are able to download the package, but not the model (which is downloaded by the module in the background). I think you are blocked from doing so.


runtime/internal/atomic: for arm and mips compare-and-swap does not have a memory barrier on failure

In the discussion about #63384 we agreed that it would be best if atomic compare-and-swap applied a memory barrier even if the comparison fails.  That avoids potential race conditions such as the one described in that issue.
Based on my own inspection, the arm and mips implementation of compare-and-swap do not implement the expected memory barrier.  The arm64 and mips64 implementations look OK.
CC @golang/arm @golang/mips @golang/runtime

@ianlancetaylor I'm not sure how to proceed with the fix for the arm platform, quoting Keith Randall reply on the CL:|

The pre-cas barriers already existed for mips and ppc64, they were not part of the fix to 63506. That said, maybe arm needs one? I'm not entirely sure why we would need a barrier both on entry and on exit. Just one of the two should suffice?
This is very tricky code. I am unsure how we would convince ourselves of the correctness...

I'm not positive this would work, but ARM does have a model checker for ARM64 memory synchronization code: https://developer.arm.com/herd7 (docs). While ARM64 is pretty different from ARM in general, I think in this particular case the code can be translated to ARM64 without loss of fidelity.


Support JsValue in Callbacks + Prevent Double Free

Allow JsValue arguments in add_callback + dead code cleanup
Prevent double free on set_property failure

Closes #109
Closes #107

Again, spurious internal quickjs error from valgrind. (need to figure out a ignore rule)


FAB ActionButton.Items are not aligning with the Action Button itself

I have my own ActionButton, but for the sake of this example ... I copied and pasted the exact example in the docs, and it is still causing the issue:
        <View style={{flex:1}}>
          <ActionButton buttonColor="rgba(231,76,60,1)">
            <ActionButton.Item buttonColor='#9b59b6' title="New Task" onPress={() => console.log("notes tapped!")}>
              <Icon name="times" style={styles.actionButtonIcon} />
            </ActionButton.Item>
            <ActionButton.Item buttonColor='#3498db' title="Notifications" onPress={() => {}}>
              <Icon name="credit-card" style={styles.actionButtonIcon} />
            </ActionButton.Item>
            <ActionButton.Item buttonColor='#1abc9c' title="All Tasks" onPress={() => {}}>
              <Icon name="envelope-o" style={styles.actionButtonIcon} />
            </ActionButton.Item>
          </ActionButton>
        </View>

And the result is as such:

Any help would be appreciated ... I know our current react native version is 0.33.1, could this be the issue?

@dekkofilms can you update this library (v2.5) and see if everything works as expected?

yeah should be fixed by now in the latest version.

okay, awesome ... i'll be back on this project tomorrow, and will try it then! thanks for the heads up.


Replace fixtures.tar.gz with regular files

In order to keep the fixtures in a form that can easily be reviewed,
this changeset puts the files from fixtures.tar.gz back into the repo as
regular files (in a separate directory, fixtures.src). In order to
prevent filenames that prevent checkouts on Windows, colons are encoded
as _@colon@_ and copied to the correct location by make.
Alternatives: files could be renamed rather than copied (using -depth
in the find command to rename files before their parent directories).
However, doing so would remove files from a location where git expects
them.

I'd say yes. Is it available on osx?
On Sat, Jun 17, 2017, 20:46 Matthias Rampke<EMAIL_ADDRESS>wrote:

@matthiasr commented on this pull request.
In Makefile
https://github.com/prometheus/procfs/pull/56#discussion_r122574475:

@@ -8,11 +8,17 @@ lint:
go get github.com/golang/lint/golint
golint *.go

-test: sysfs/fixtures/.unpacked
+test: sysfs/fixtures/.copied
Working on it. The straightforward way will require GNU make, is that
acceptable?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/procfs/pull/56#discussion_r122574475, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAANaLRaEb8tX8YfohO4n8kgC8g7yvReks5sFB7fgaJpZM4N5gJl
.

Is it available on osx?

Yes, it's the default there. It's not on other BSDs, but can be installed.

I agree. Renaming is a non-starter, destroying the checkout all the time
will make development impossible. Let's stick with the approach in this PR
but use a phony target instead of the .copied file
On Sun, Jun 18, 2017, 17:48 Tobias Schmidt<EMAIL_ADDRESS>wrote:

@grobie commented on this pull request.
In Makefile
https://github.com/prometheus/procfs/pull/56#discussion_r122594213:

@@ -8,11 +8,17 @@ lint:
go get github.com/golang/lint/golint
golint *.go

-test: sysfs/fixtures/.unpacked
+test: sysfs/fixtures/.copied
The problem is, the marker needs to be somehow dependent on the files in
source code. Otherwise the copy command will never be executed again.
That's what I asked @matthiasr https://github.com/matthiasr to help
with, but apparently it's not possible due to Make's lack of colon support
in filenames.
I'd prefer slower test execution over stale fixture files. I'd like to
prevent developers from unnecessary debug sessions when they forgot to
remove the marker file manually to trigger an update.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/procfs/pull/56#discussion_r122594213, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAICBiPD1Lag8jnxnd7ow6E9AJDCsWhQks5sFUa_gaJpZM4N5gJl
.

Thank you @ideaship!

Sigh @ #57. I need to find a Windows box for testing.


ipython issue

Hi, when I ran bcbio with ipython parallel, I found that the program just stuck in the following log stage,
[2017-05-08T15:25Z] ica1: System YAML configuration: /path/to/bcbio/galaxy/bcbio_system.yaml
[2017-05-08T15:25Z] ica1: Resource requests: bwa, sambamba, samtools; memory: 3.00, 3.00, 3.00; cores: 16, 16, 16
[2017-05-08T15:25Z] ica1: Configuring 13 jobs to run, using 16 cores each with 48.1g of memory reserved for each job
[ProfileCreate] Generating default config file: u'/path/to/work/log/ipython/ipython_config.py'
[ProfileCreate] Generating default config file: u'/path/to/work/log/ipython/ipython_kernel_config.py'
[ProfileCreate] Generating default config file: u'/path/to/work/log/ipython/ipcontroller_config.py'
[ProfileCreate] Generating default config file: u'/path/to/work/log/ipython/ipengine_config.py'
[ProfileCreate] Generating default config file: u'/path/to/work/log/ipython/ipcluster_config.py'
2017-05-08 10:25:35.862 [IPClusterStart] Using existing profile dir: u'/path/to/work/log/ipython'
2017-05-08 10:25:35.862 [IPClusterStart] Searching path [u'/path/to/work', u'/path/to/work/log/ipython', '/usr/local/etc/ipython', '/etc/ipython'] for config files
2017-05-08 10:25:35.862 [IPClusterStart] Attempting to load config file: ipython_config.py
2017-05-08 10:25:35.862 [IPClusterStart] Looking for ipython_config in /etc/ipython
2017-05-08 10:25:35.863 [IPClusterStart] Looking for ipython_config in /usr/local/etc/ipython
2017-05-08 10:25:35.863 [IPClusterStart] Looking for ipython_config in /path/to/work/log/ipython
2017-05-08 10:25:35.863 [IPClusterStart] Loaded config file: /path/to/log/ipython/ipython_config.py
2017-05-08 10:25:35.863 [IPClusterStart] Looking for ipython_config in /path/to/work
2017-05-08 10:25:35.864 [IPClusterStart] Attempting to load config file: ipcluster_9181c4bb_63a3_4420_9882_dc45721b936a_config.py
2017-05-08 10:25:35.864 [IPClusterStart] Looking for ipcontroller_config in /etc/ipython
2017-05-08 10:25:35.864 [IPClusterStart] Looking for ipcontroller_config in /usr/local/etc/ipython
2017-05-08 10:25:35.864 [IPClusterStart] Looking for ipcontroller_config in /path/to/work/log/ipython
2017-05-08 10:25:35.865 [IPClusterStart] Loaded config file: /path/to/work/log/ipython/ipcontroller_config.py
2017-05-08 10:25:35.865 [IPClusterStart] Looking for ipcontroller_config in /path/to/work
2017-05-08 10:25:35.865 [IPClusterStart] Attempting to load config file: ipcluster_9181c4bb_63a3_4420_9882_dc45721b936a_config.py
2017-05-08 10:25:35.866 [IPClusterStart] Looking for ipengine_config in /etc/ipython
2017-05-08 10:25:35.866 [IPClusterStart] Looking for ipengine_config in /usr/local/etc/ipython
2017-05-08 10:25:35.866 [IPClusterStart] Looking for ipengine_config in /path/to/work/log/ipython
2017-05-08 10:25:35.866 [IPClusterStart] Loaded config file: /path/to/log/ipython/ipengine_config.py
2017-05-08 10:25:35.866 [IPClusterStart] Looking for ipengine_config in /path/to/work
2017-05-08 10:25:35.867 [IPClusterStart] Attempting to load config file: ipcluster_9181c4bb_63a3_4420_9882_dc45721b936a_config.py
2017-05-08 10:25:35.867 [IPClusterStart] Looking for ipcluster_config in /etc/ipython
2017-05-08 10:25:35.867 [IPClusterStart] Looking for ipcluster_config in /usr/local/etc/ipython
2017-05-08 10:25:35.867 [IPClusterStart] Looking for ipcluster_config in /path/to/work/log/ipython
2017-05-08 10:25:35.868 [IPClusterStart] Loaded config file: /path/to/work/log/ipython/ipcluster_config.py
2017-05-08 10:25:35.868 [IPClusterStart] Looking for ipcluster_config in /path/to/work
After further checking, I found the following error in /path/to/work/log/ipython/log/ipcluster-9337b06d-1b2d-4fd9-b7fb-14c0833bca12-23613.log,
2017-05-08 12:48:13.941 [IPClusterStart] Starting ipcluster with [daemon=True]
2017-05-08 12:48:13.946 [IPClusterStart] Creating pid file: /path/to/work/log/ipython/pid/ipcluster-9337b06d-1b2d-4fd9-b7fb-14c0833bca12.pid
2017-05-08 12:48:13.947 [IPClusterStart] Starting Controller with cluster_helper.cluster.BcbioSLURMControllerLauncher
2017-05-08 12:48:13.947 [IPClusterStart] Starting BcbioSLURMControllerLauncher: ['sbatch', u'./SLURM_controller29aef5c8-bd49-4137-b6bd-3d8b4100ed59']
2017-05-08 12:48:13.948 [IPClusterStart] adding PBS queue settings to batch script
2017-05-08 12:48:13.948 [IPClusterStart] Writing batch script: ./SLURM_controller29aef5c8-bd49-4137-b6bd-3d8b4100ed59
2017-05-08 12:48:13.959 [IPClusterStart] ERROR | Controller start failed
Traceback (most recent call last):
File "/path/to/bcbio/anaconda/lib/python2.7/site-packages/ipyparallel/apps/ipclusterapp.py", line 503, in start_controller
self.controller_launcher.start()
File "/path/to/bcbio/anaconda/lib/python2.7/site-packages/cluster_helper/cluster.py", line 506, in start
return super(BcbioSLURMControllerLauncher, self).start(1)
File "/path/to/bcbio/anaconda/lib/python2.7/site-packages/ipyparallel/apps/launcher.py", line 1151, in start
output = check_output(self.args, env=os.environ)
File "/path/to/bcbio/anaconda/lib/python2.7/subprocess.py", line 574, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['sbatch', u'./SLURM_controller29aef5c8-bd49-4137-b6bd-3d8b4100ed59']' returned non-zero exit status 1
From the strace, it seems that it’s trying to open up configuration file that doesn’t exist:
stat("/path/to/work/log/ipython/security/ipcontroller-9181c4bb-63a3-4420-9882-dc45721b936a-client.json", 0x7ffd41900600) = -1 ENOENT (No such file or directory)
The program is looping through the same section again and again
Could you help me with debugging?
Thanks!

Sorry about the issue and thanks for the detailed report. It appears as if the created batch script is not working on your cluster for some reason. My suggestion would be to run:
sbatch ./SLURM_controller29aef5c8-bd49-4137-b6bd-3d8b4100ed5

outside of bcbio and it'll hopefully give a useful message about what it's failing. If you're still stuck after seeing this error happy to provide more specific information about what might be happening. Hope this helps.

It's still stuck and I found the following info in /nebraska/inbre/home/peng.xiao/cowan/26th_s/work/log/ipython/log/ipcontroller-c399aec8-9400-4acb-9382-fb5a96843a88-35115.log, and it keeps going.
 What did the sbatch command above report when you submitted it outside of bcbio? That should be the diagnostic error to help debug what is going wrong. Thanks much.

Hi Brad,
What do you mean to submit the sbatch outside of bcbio? All the slurms were submitted under work folder, which is different from the bcbio folder. First, in work folder, I submitted
sbatch myjob.slurm
The running was stuck without showing any error in bcbio-nextgen*.log files but found the following error in /path/to/work/log/ipython/log/ipcluster-c399aec8-9400-4acb-9382-fb5a96843a88-35040.log,
2017-05-10 09:26:52.476 [IPClusterStart] Starting ipcluster with [daemon=True]
2017-05-10 09:26:52.481 [IPClusterStart] Creating pid file: /path/to/work/log/ipython/pid/ipcluster-c399aec8-9400-4acb-9382-fb5a96843a88.pid
2017-05-10 09:26:52.482 [IPClusterStart] Starting Controller with cluster_helper.cluster.BcbioSLURMControllerLauncher
2017-05-10 09:26:52.482 [IPClusterStart] Starting BcbioSLURMControllerLauncher: ['sbatch', u'./SLURM_controller8bc7e8ee-7cc8-49d0-b91c-324341a04969']
2017-05-10 09:26:52.483 [IPClusterStart] adding PBS queue settings to batch script
2017-05-10 09:26:52.483 [IPClusterStart] Writing batch script: ./SLURM_controller8bc7e8ee-7cc8-49d0-b91c-324341a04969
2017-05-10 09:26:52.495 [IPClusterStart] ERROR | Controller start failed
Traceback (most recent call last):
File "/path/to/bcbio/anaconda/lib/python2.7/site-packages/ipyparallel/apps/ipclusterapp.py", line 503, in start_controller
self.controller_launcher.start()
File "/path/to/bcbio/anaconda/lib/python2.7/site-packages/cluster_helper/cluster.py", line 506, in start
return super(BcbioSLURMControllerLauncher, self).start(1)
File "/path/to/bcbio/anaconda/lib/python2.7/site-packages/ipyparallel/apps/launcher.py", line 1151, in start
output = check_output(self.args, env=os.environ)
File "/path/to/bcbio/anaconda/lib/python2.7/subprocess.py", line 574, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['sbatch', u'./SLURM_controller8bc7e8ee-7cc8-49d0-b91c-324341a04969']' returned non-zero exit status 1
Then under the work same folder, I submitted
sbatch SLURM_controller8bc7e8ee-7cc8-49d0-b91c-324341a04969
The running is still stuck and no errors in error in bcbio-nextgen*.log files but I found a new /path/to/work/log/ipython/log/ipcontroller-c399aec8-9400-4acb-9382-fb5a96843a88-35115.log, which showed the info that I showed in the previous message and is stilling adding with stopping.
Thanks!

Sorry for the confusion, I meant what does the command:
sbatch SLURM_controller8bc7e8ee-7cc8-49d0-b91c-324341a04969

show? Is there an error? Does the bcbio controller job get started correctly if you look at the SLURM queue? To be clear, I don't expect this will rescue your previous job, just that this command returned an error previously so we need to figure out what it is. Generally, I suspect there is something in the SLURM_controller* file that your cluster doesn't like that is causing it to error out.
If it works cleanly, which it seems like it might based on the log file, maybe there is something wrong with your PATH that sbatch is not available when running with bcbio? Are you submitting the main bcbio_nextgen.py to a worker node which perhaps has a different path or permissions to run sbatch? Hope this helps some.

the slurm queue shows the status RUNNING.
JOBID  PARTITION                                NAME CPUS       START_TIME         USER    STATE       TIME TIME_LIMIT  NODES NODELIST(REASON)
7834      inbre                      project_s-c 2          2017-05-10    peng.xiao  RUNNING       5:25 7-00:00:00      1 ica1
7833      inbre                        project_s 2          2017-05-10    peng.xiao  RUNNING       6:38 7-00:00:00      1 ica1
No error showed except the content that I showed previously.

Thanks for confirming, it sounds like the batch script is okay but something is going wrong with the submission based on the host where it gets submitted from. Is the machine you're testing the manual sbatch on the same as the machine where bcbio submits it during the run? Is it possible there are machine specific differences causing it to work in one place and not the other. Sorry to not have a definite answer but hope this helps some for debugging.

Thank you Brad! You provided important clues and the problem has been solved.


Bugfix: Hanging of websocket task on HTTPStatus error exception

Hopefully fixes https://github.com/py-smart-gardena/hass-gardena-smart-system/issues/121

Concerning the exceptions:
I think it would be best if the Gardena integration removes any dependencies to the OAUTH2 things like here: https://github.com/py-smart-gardena/hass-gardena-smart-system/blob/6d76239c81ed622fb027f1711173d747f85b30d2/custom_components/gardena_smart_system/init.py#L14
And instead defines its own exceptions.
Furthermore, in the integration there is no try/catch block surrounding this: https://github.com/py-smart-gardena/hass-gardena-smart-system/blob/6d76239c81ed622fb027f1711173d747f85b30d2/custom_components/gardena_smart_system/init.py#L62
With this PR it is not really necessary as the websaocket task handles them and automatically restarts things. But maybe we want to define again our exception that we could raise after having retried for some timeout without success.

Thx for the PR ! By reading it it should solve one edge case (but not solve the root of the 429 issues i guess .. )
I will try it tonight and I also have an idea for the source of these 429. I ll try it tonight also !
Thx again !
I may have

My guess is that you cannot really influence the 429 issue.
It just happens from time to time. Maybe the Gardena people have just a limited number of resources available.
The only thing I could think would be that we would have to reuse the websocket connection URL again after it is being closed.
But then again, I read that the URL is only valid for a few seconds. So when the connection was established for 2h I think it wonÄt be valid anymore. As a result we would have to contact their Gardena at least every two hours.
I only get the 429 from the Gardena API. So far I never got it from the authentciation API.

Just happened again:
device update: <gardena.devices.sensor.Sensor object at 0x7ff2480fa910>
2022-07-21 19:18:27 DEBUG    Closing websocket ..
2022-07-21 19:18:27 DEBUG    Sleeping 10 seconds ..
2022-07-21 19:18:37 DEBUG    Trying to connect to gardena API....
2022-07-21 19:18:37 DEBUG    Trying to get Websocket url
oauth_client.py: ensure_active_token()
2022-07-21 19:18:37 DEBUG    HTTP Request: POST https://api.smart.gardena.dev/v1/websocket "HTTP/1.1 429 "
2022-07-21 19:18:37 DEBUG    Websocket url: got response
2022-07-21 19:18:37 DEBUG    HTTPStatusError ..
2022-07-21 19:18:37 DEBUG    Closing websocket ..
2022-07-21 19:18:37 DEBUG    Sleeping 10 seconds ..
2022-07-21 19:18:47 DEBUG    Trying to connect to gardena API....
2022-07-21 19:18:47 DEBUG    Trying to get Websocket url
oauth_client.py: ensure_active_token()
2022-07-21 19:18:47 DEBUG    HTTP Request: POST https://api.smart.gardena.dev/v1/websocket "HTTP/1.1 201 Created"
2022-07-21 19:18:47 DEBUG    Websocket url: got response
2022-07-21 19:18:47 DEBUG    Websocket url retrieved successfully
2022-07-21 19:18:47 DEBUG    Connecting to websocket ..
2022-07-21 19:18:48 DEBUG    Connected !
device update: <gardena.devices.sensor.Sensor object at 0x7ff24896cfa0>

Maybe another thing: what about the websocket ping interval?

we recommend sending ping messages every 150 seconds to keep the connection open

Yes, that was my guess.
I think this is the sole request we are doing and doign it every 10 seconds.
I was planning to change that tonight ;)

429 codes means client errors, the api is not responding.
I think we have a behaviour wich request the API too often, and i only see the ping ..

Please also read this interesting page:
https://www.useanvil.com/blog/engineering/throttling-and-consuming-apis-with-429-rate-limits/
Do we also receive a „retry-after“ http header field?
This eould easily solve it.

The approach merged here will, instead of backing of in case of a 429 Limit Exceeded error, hammer the server forever. Which will most likely make it block forever. So now the application will be broken more or less totally, until killed and left idle for some time (assuming the APIs rate limits works as expected) :)
See this section:
https://developer.husqvarnagroup.cloud/apis/gardena-smart-system-api#rate-limits

An application that behaves as a good citizen of the Gardena eco system should not exceed the following limits:

On average one call every fifteen minutes.
3'000 requests per month.

If there is no Retry-After header in case of 429, I would sleep N=2 seconds or so, then retry. Then if it fails with 429 again, incresae the wait time a bit for each failure, until successful call (in which case back-off value is reset, until we get 429 in future).

Also, perhaps build it into lower-level functions such as __call_smart_system_get and call_smart_system_service.

I agree, we should have a backoff timeout in case of 429 !
I am looking at the possibility to implement suche a feature.

Worth mentioning, the recommendations specified by Gardena does not reflect the way it's actually working.
Today I got a 429 on the websocket call, at 22:32:57... The only previous call was another websocket call at 20:32:46 (and before that 18:32, 16:32,14:32, all which gave 201 Created).. A lot fewer than "one call per every 15 minutes", and since the subsequent call 15 seconds later (my backoff logic) passed fine, probably not near the 3k/req month either.

(my backoff logic) passed fine

Could you open another PR which adds the backoff logic for all?

(my backoff logic) passed fine

Could you open another PR which adds the backoff logic for all?

My "backoff logic" is more of a hack: catch those exceptions outside of my (small) program's main loop and just restart the whole thing. So not really something that can be PR'd.
For the record, it looks like this (quick evening hack just to get things recovering automatically):
backoff = 10
while True:
    try:
        asyncio.run(main())
        break
    except httpx.HTTPStatusError as e:
        if e.response.status_code != 429:
            raise
        logging.exception('HTTPStatusError')
    except Exception as e:
        if e.args[0] != "Limit Exceeded":
            raise
        logging.exception('anon exec')

    logging.warning("Gardena API limit exceeded.. global retrying in %d...", backoff)
    time.sleep(backoff)
    backoff = min(backoff * 1.5, 3600)

oh and then inside main(), after first successful call I set backoff = 10.
Did I say hacky? :see_no_evil:

Hello,
I just implement a backoff mechanism in the library yesterday in the beta channel (b906). It may be a little aggressive for now but it is less that it is now ;)

It is implmeented only on get for data, on uthentication and on websocket url retrieval.
I am not sure I want to have this mechanisme for service calls (e.g. start mower, watering ...)

If the on_exception decorator can be made to filter on only 429's (as in their giveup=fatal_code example perhaps) , I'd say that looks like a good solution. perhaps also for 503?
Unsure about PUTs..
For a  404 or 400 or something else, I don't think it should retry for any kind of calls.

I am not sure I want to have this mechanism for service calls (e.g. start mower, watering ...)

I would not want to have this as it seems dangerous to me. I do not want to start the mower at an unknown time in the future when the conditions to start it are not known anymore. For example.
Instead I would do like this dev here: https://github.com/ekutner/home-connect-hass
He connect to the Bosch/Siemens REST API for Home Connect devices. They also have rate limits.
He added a dedicated device for the REST API which gives a status:

INIT - The integration is initializing
RUNNING - The integration has started running
LOADED - The integration finished loading the initial data from the Home Connect service
READY - The integration has successfully subscribed to real time updates from the cloud service and is now fully functional.
BLOCKED - The Home Connect service has blockde additional API calls for a period of time due to exceeding the service rate limits It may take up to one minute to go from LOADED to READY

This would be perfectly well here as well.
If blocked the user has to decided in his/her automation what to do.

BTW: He is lucky as he gets a "retry-after" header: https://github.com/ekutner/home-connect-async/blob/a907d3c2ca39e96610f66cfabada9b868663567a/home_connect_async/api.py#L85

Would it make sense to add retry for other errors too?
My python code had crashed:
Traceback (most recent call last):
  File "test.py", line 303, in <module>
    asyncio.run(main())
  File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "test.py", line 253, in main
    await smart_system.start_ws(location)
  File "/usr/home/johan/dev/gardena-async/py-smart-gardena/src/gardena/smart_system.py", line 189, in start_ws
    ws_url = await self.__get_ws_url(location)
  File "/usr/home/johan/dev/gardena-async/venv/lib/python3.7/site-packages/backoff/_async.py", line 151, in retry
    ret = await target(*args, **kwargs)
  File "/usr/home/johan/dev/gardena-async/py-smart-gardena/src/gardena/smart_system.py", line 216, in __get_ws_url
    data=json.dumps(args, ensure_ascii=False),
  File "/usr/home/johan/dev/gardena-async/venv/lib/python3.7/site-packages/httpx/_client.py", line 1855, in post
    extensions=extensions,
  File "/usr/home/johan/dev/gardena-async/venv/lib/python3.7/site-packages/authlib/integrations/httpx_client/oauth2_client.py", line 92, in request
    method, url, auth=auth, **kwargs)
  File "/usr/home/johan/dev/gardena-async/venv/lib/python3.7/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/home/johan/dev/gardena-async/venv/lib/python3.7/site-packages/httpx/_client.py", line 1618, in send
    history=[],
  File "/usr/home/johan/dev/gardena-async/venv/lib/python3.7/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    history=history,
  File "/usr/home/johan/dev/gardena-async/venv/lib/python3.7/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/home/johan/dev/gardena-async/venv/lib/python3.7/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/home/johan/dev/gardena-async/venv/lib/python3.7/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/home/johan/dev/gardena-async/venv/lib/python3.7/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout


Swin-L config without Objects365 pretraining and Objects365 pretraining setting

Hello,
Thank you for sharing the great work! We really find it useful.
We have one question regarding how to train your model using Swin-L, without Objects365 pretraining (i.e, only ImageNet-21K pretraining). Do you mind sharing the config or any settings for us to try?
Additionally, the script to pretrain the Swin-L on Objects365 (deta_swin_pre.sh) is missing. Is it also possible that you can share that script and Objects365's setting (e.g., all images are used? and so on?).
Thanks,

Hi Jay,
I'm glad you like the work!

Swin-L without Objects365 pretraining: we have not tried this, but I think you can just replace the Objects365 checkpoint with the IN21K checkpoint from https://github.com/microsoft/Swin-Transformer
the script to pretrain the Swin-L on Objects365: We trained on Objects365 training set (no merge with val) on 24epoch schedule (details in the paper). The only change to the code are increasing num_classes and training with higher resolution images. Happy to answer any other questions I can.

Awesome. Thank you, Jeffrey, for the quick response and for the clarification.
It is clear to me and we will run the experiments as suggested and see how it goes.
Thanks,


Can't use pocketbase on React Native.

Hey!
I tried to use pocketbase with React Native and get error:
ClientResponseError 0: Something went wrong while processing your request.
Can someone please confirm that pocketbase is not working on React Native and it's possible to add support for React Native too?

@niputy I'm closing the issue since I'm not able to reproduce it (tested with Expo). Please provide the full error stack trace and/or a code sample how you initialize the SDK and what method calls you are making.

Looks like it's http issue described here. console.log(err.originalError) show TypeError: Network request failed
I used latest versions of everything, you actually can test on my app here.
It's possible to change pocketbase url from http to https?

It's possible to change pocketbase url from http to https?

You can start the PocketBase executable with --https="yourdomain.com:443" but it'll issue a Let's encrypt certificate and expects a valid public domain (it will not work with bare IPs).
It is hard to debug what exactly is the issue since the error is too generic and there is no stack trace attached. Make sure that the your device have access to the PocketBase url address (in the code it is <IP_ADDRESS> but if you are running inside an emulator you have to use <IP_ADDRESS> per https://developer.android.com/studio/run/emulator-networking#networkaddresses).

Omg, I just change <IP_ADDRESS> to <IP_ADDRESS> and now it's working. Thank you very much!
Issue was resolved.

I know its been a while but if anyone comes across this issue you can use ngrok.
ngrok http 8090
# Or whatever port you have pocketbase set to.

note: you do have to make an account but it is free.

@ganigeorgiev , so it's not possible to have realtime in react-native?

@NoCluePS I don't know how did you come to this conclusion.
It should work fine but you'll have to load a polyfill (see the note in https://github.com/pocketbase/js-sdk#nodejs-via-npm). Also if you are testing on Android in debug mode please check the linked resources in https://github.com/pocketbase/pocketbase/discussions/2484#discussioncomment-6118523.
If you are still not able to make it work, feel free to open a Q&A discussion in the main repo with more details about your setup and how are you testing and I'll try to investigate it.

Sorry, didn't refresh hard enough apparently :D


Add evidence being reported

Example of evidence so far, but is incomplete. Will likely want to do broad strokes in the main spec and then details in the ROM and Runtime specs.

Component
Type
Evidence
Measurer
Evidence Reporting
Measurement Reporting
Validity Period

Caliptra
Hardware State
Lifecycle state
Caliptra ROM
FMC.DiceTcbInfo.OperationalFlags
PCR0 & PCR1
Until next Power-Good

Debug lock state
Caliptra ROM
FMC.DiceTcbInfo.OperationalFlags
PCR0 & PCR1
Until next Power-Good

ROM Policy Enforcement
Anti-rollback-disable state
Caliptra ROM
FMC.DiceTcbInfo.OperationalFlags
PCR0 & PCR1
Until next Power-Good

Vendor pub key hash
Caliptra ROM
FMC.DiceTcbInfo.FWID
PCR0 & PCR1
Until next Power-Good

Owner pub key hash
Caliptra ROM
FMC.DiceTcbInfo.FWID
PCR0 & PCR1
Until next Power-Good

Vendor ECC pub key idx
Caliptra ROM
FMC.DiceTcbInfo.FWID
PCR0 & PCR1
Until next Power-Good

Min SVN
Caliptra ROM
FMC.DiceTcbInfo.Svn
PCR0 & PCR1
Until next Power-Good

Running Firmware
FMC hash
Caliptra ROM
FMC.DiceTcbInfo.FWID
PCR0 & PCR1
Until next Power-Good

SVN
Caliptra ROM
FMC.DiceTcbInfo.Svn
PCR0 & PCR1
Until next Power-Good

RT hash
Caliptra FMC
RT.DiceTcbInfo.FWID.JourneyRT.DiceTcbInfo.FWID.CurrentRTU Event Log

Until next RT Update

SoC Manager
Hardware State
ROM patches
SoC Manager ROM

Until next Power-Good

ROM Policy Enforcement

SoC Manager ROM

Until next Power-Good

Running Firmware
SoC Manager FMC hash
SoC Manager ROM

Until next Power-Good

SoC Manager RT hash
SoC Manager FMC

Until next RT hitless update

SoC Application CPUs
Running Firmware
Bootstrap hash
SoC Manager RT

Until next CPU reset

App CPU firmware hash
Bootstrap

Until next CPU reset or App CPU firmware hitless update

UEFI hash
App CPU firmware

Until next CPU reset or UEFI hitless update

Realm manager hash
SoC Manager RT

Until next CPU reset or Realm manager hitless update

Kirkland
Metadata
TPM identity pub key hash
UEFI

-
Until next CPU reset

Varun is there more work needed? Can we complete by 9/8 and then hand off to steph-morton, please.


POST request for adding an observation

It looks like the documentation for adding an observation is incomplete. Could you please help me figure out the proper format of the URL?
Namely:

The example url has a parameter named "name" while the online form request a "scientific name". Therefore, it is not obvious how each of the fields in the form should be named in the url. Could you provide a complete mapping?

For fields like "Used references" it is unclear what should be sent. Both a string and a boolean if checked? Just a string?

The readme refers to a file script/test_api to see how images should be encoded. However it seems that this file doesn't exist.

I've made a few requests using my key in this url:

https://mushroomobserver.org/api2/observations?api_key=xxx&name=Agaricus&location=Pasadena,+California,+USA&date=2016-08-06&notes=growing+in+lawn

I got a 200 on each of them. However I don't see them when I go to the my observations page.
Thank you in advance for you help.
Best regards.

Hi there,
The variable "name" in a POST request to observations is the scientific
name.  To get a full list of parameters do:
curl -X POST -F api_key=___ -F help=1
https://mushroomobserver.org/api2/observations
Here they are in the order they appear in the online form, with some extra
help notes:
date: date (YYYY-MM-DD)
location: place_name (e.g., "Pasadena, California, USA")
is_collection_location: boolean (default, is this location where mushroom
was found?)
latitude: latitude
longitude: longitude
altitude: altitude
gps_hidden: boolean (default=false, hide exact coordinates?)
name: name (default=Fungi Bartl.)
vote: float (default 3="I'd call it that", 2="probably", 1="could be", etc.)
reason_1: string (recognized by sight, just send a string, if you include
this parameter it's the same as checking the box, even if you pass an empty
string)
reason_2: string (used references)
reason_3: string (micro features)
reason_4: string (chemical features)
has_specimen: boolean
collectors_name: string
collection_number: string
herbarium: herbarium
accession_number: string (unique fungarium id)
notes: string
notes[$field]: string (set value of the custom notes template field,
substitute field name for "$field")
Other parameters you probably won't use:
images: image list (ids of images you have already uploaded, it's easier to
create the obs first then upload images one at a time attaching each one to
the obs as you go)
thumbnail: image (id of one of the images you have already attached, this
is really only used in PATCH requests)
species_lists: species_list list (id, must have edit permission)
projects: project list (id, must be member)
initial_det: string (ignore this)
source: enum (limit=mo_android_app|mo_api|mo_iphone_app|mo_website,
default=mo_api)
log: boolean (default, log this action on main page activity log and RSS
feed?)
If sorry, yes, it looks like script/test_api has disappeared.
It looks like you can do uploads like this:
curl -X POST -F @.*** -F etc.
https://mushroomobserver.org/api2/images
Let me know if that doesn't work.
I tried your test URL verbatim (replacing xxx with my api key) and it
worked just fine.  Can you give me your user id, and tell me approximately
when you tried it, and I'll look it up in the log to see if I can figure
out what went awry.
Did I miss anything?
-Jason
On Thu, Aug 17, 2023 at 1:54 AM fran-aubry @.***> wrote:

It looks like the documentation for adding an observation is incomplete.
Could you please help me figure out the proper format of the URL?
Namely:

The example url has a parameter named "name" while the online form
https://mushroomobserver.org/observations/new request a "scientific
name". Therefore, it is not obvious how each of the fields in the form
should be named in the url. Could you provide a complete mapping?
2.
For fields like "Used references" it is unclear what should be sent.
Both a string and a boolean if checked? Just a string?
3.
The readme refers to a file script/test_api to see how images should
be encoded. However it seems that this file doesn't exist.
4.
I've made a few requests using my key in this url:
https://mushroomobserver.org/api2/observations?api_key=xxx&name=Agaricus&location=Pasadena,+California,+USA&date=2016-08-06&notes=growing+in+lawn
I got a 200 on each of them. However I don't see them when I go to the my
observations page.
Thank you in advance for you help.
Best regards.
—
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/issues/1558, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAYTNNMIKEPNU6EBVIDCL73XVWWZDANCNFSM6AAAAAA3TRHENE
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Hey, thank you for the reply.
I made this request today at around 11:02 (UTC+8)
https://mushroomobserver.org/api2/observations?api_key=****&date=2023-08-18&place_name=Earth&latitude=20.401271959729456&longitude=-3.33984375&altitude=288.0030212402344&gps_hidden=false&vote=0&reason_1=Yes
I got a 200 response but it doesn't show in my observation list. Could you
take a look?
Also, is there a proper way to test these requests? I don't want to pollute
the database while testing.
On Thu, Aug 17, 2023 at 2:44 PM Jason Hollinger @.***>
wrote:

Hi there,
The variable "name" in a POST request to observations is the scientific
name. To get a full list of parameters do:
curl -X POST -F api_key=___ -F help=1
https://mushroomobserver.org/api2/observations
Here they are in the order they appear in the online form, with some extra
help notes:
date: date (YYYY-MM-DD)
location: place_name (e.g., "Pasadena, California, USA")
is_collection_location: boolean (default, is this location where mushroom
was found?)
latitude: latitude
longitude: longitude
altitude: altitude
gps_hidden: boolean (default=false, hide exact coordinates?)
name: name (default=Fungi Bartl.)
vote: float (default 3="I'd call it that", 2="probably", 1="could be",
etc.)
reason_1: string (recognized by sight, just send a string, if you include
this parameter it's the same as checking the box, even if you pass an
empty
string)
reason_2: string (used references)
reason_3: string (micro features)
reason_4: string (chemical features)
has_specimen: boolean
collectors_name: string
collection_number: string
herbarium: herbarium
accession_number: string (unique fungarium id)
notes: string
notes[$field]: string (set value of the custom notes template field,
substitute field name for "$field")
Other parameters you probably won't use:
images: image list (ids of images you have already uploaded, it's easier
to
create the obs first then upload images one at a time attaching each one
to
the obs as you go)
thumbnail: image (id of one of the images you have already attached, this
is really only used in PATCH requests)
species_lists: species_list list (id, must have edit permission)
projects: project list (id, must be member)
initial_det: string (ignore this)
source: enum (limit=mo_android_app|mo_api|mo_iphone_app|mo_website,
default=mo_api)
log: boolean (default, log this action on main page activity log and RSS
feed?)
If sorry, yes, it looks like script/test_api has disappeared.
It looks like you can do uploads like this:
curl -X POST -F @.*** -F etc.
https://mushroomobserver.org/api2/images
Let me know if that doesn't work.
I tried your test URL verbatim (replacing xxx with my api key) and it
worked just fine. Can you give me your user id, and tell me approximately
when you tried it, and I'll look it up in the log to see if I can figure
out what went awry.
Did I miss anything?
-Jason
On Thu, Aug 17, 2023 at 1:54 AM fran-aubry @.***> wrote:

It looks like the documentation for adding an observation is incomplete.
Could you please help me figure out the proper format of the URL?
Namely:

The example url has a parameter named "name" while the online form
https://mushroomobserver.org/observations/new request a "scientific
name". Therefore, it is not obvious how each of the fields in the form
should be named in the url. Could you provide a complete mapping?
2.
For fields like "Used references" it is unclear what should be sent.
Both a string and a boolean if checked? Just a string?
3.
The readme refers to a file script/test_api to see how images should
be encoded. However it seems that this file doesn't exist.
4.
I've made a few requests using my key in this url:

https://mushroomobserver.org/api2/observations?api_key=xxx&name=Agaricus&location=Pasadena,+California,+USA&date=2016-08-06&notes=growing+in+lawn

I got a 200 on each of them. However I don't see them when I go to the
my
observations page.
Thank you in advance for you help.
Best regards.
—
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/issues/1558, or
unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AAYTNNMIKEPNU6EBVIDCL73XVWWZDANCNFSM6AAAAAA3TRHENE>

.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

—
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1681714192,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA2CLA6YXYM6OVQR6EVT4MDXVW4UHANCNFSM6AAAAAA3TRHENE
.
You are receiving this because you authored the thread.Message ID:
@.***>

Hi again,
For some reason your requests are coming through as HEAD requests, not
POSTs.
HEAD "/api2/observations" for <IP_ADDRESS> at 2023-08-18 03:10:53 +0000
HEAD "/observations/new" for <IP_ADDRESS> at 2023-08-18 03:10:55 +0000
HEAD "/api2/images" for <IP_ADDRESS> at 2023-08-18 03:10:55 +0000
HEAD
"/api2/observations?api_key=****&date=2023-08-18&place_name=Earth&latitude=20.401271959729456&longitude=-3.33984375&altitude=288.0030212402344&gps_hidden=false&vote=0&reason_1=Yes"
for <IP_ADDRESS> at 2023-08-18 03:10:57 +0000
HEAD
"/api2/observations?api_key=xxx&name=Agaricus&location=Pasadena,+California,+USA&date=2016-08-06&notes=growing+in+lawn"
for <IP_ADDRESS> at 2023-08-18 03:10:58 +0000
HEAD "/account/login/new" for <IP_ADDRESS> at 2023-08-18 03:10:58 +0000
That's all of the requests from that IP today.
I wouldn't worry overly about polluting the database.  You can only create
one object at a time.  Just test your code on a single observation at a
time until it looks like all the kinks are worked out, then let it loose.
You can easily delete anything on the website as necessary.  And more to
the point, there isn't any other option!
-Jason
On Thu, Aug 17, 2023 at 11:06 PM fran-aubry @.***>
wrote:

Hey, thank you for the reply.
I made this request today at around 11:02 (UTC+8)
https://mushroomobserver.org/api2/observations?api_key=****&date=2023-08-18&place_name=Earth&latitude=20.401271959729456&longitude=-3.33984375&altitude=288.0030212402344&gps_hidden=false&vote=0&reason_1=Yes
I got a 200 response but it doesn't show in my observation list. Could you
take a look?
Also, is there a proper way to test these requests? I don't want to
pollute
the database while testing.
On Thu, Aug 17, 2023 at 2:44 PM Jason Hollinger @.***>
wrote:

Hi there,
The variable "name" in a POST request to observations is the scientific
name. To get a full list of parameters do:
curl -X POST -F api_key=___ -F help=1
https://mushroomobserver.org/api2/observations
Here they are in the order they appear in the online form, with some
extra
help notes:
date: date (YYYY-MM-DD)
location: place_name (e.g., "Pasadena, California, USA")
is_collection_location: boolean (default, is this location where
mushroom
was found?)
latitude: latitude
longitude: longitude
altitude: altitude
gps_hidden: boolean (default=false, hide exact coordinates?)
name: name (default=Fungi Bartl.)
vote: float (default 3="I'd call it that", 2="probably", 1="could be",
etc.)
reason_1: string (recognized by sight, just send a string, if you
include
this parameter it's the same as checking the box, even if you pass an
empty
string)
reason_2: string (used references)
reason_3: string (micro features)
reason_4: string (chemical features)
has_specimen: boolean
collectors_name: string
collection_number: string
herbarium: herbarium
accession_number: string (unique fungarium id)
notes: string
notes[$field]: string (set value of the custom notes template field,
substitute field name for "$field")
Other parameters you probably won't use:
images: image list (ids of images you have already uploaded, it's easier
to
create the obs first then upload images one at a time attaching each one
to
the obs as you go)
thumbnail: image (id of one of the images you have already attached,
this
is really only used in PATCH requests)
species_lists: species_list list (id, must have edit permission)
projects: project list (id, must be member)
initial_det: string (ignore this)
source: enum (limit=mo_android_app|mo_api|mo_iphone_app|mo_website,
default=mo_api)
log: boolean (default, log this action on main page activity log and RSS
feed?)
If sorry, yes, it looks like script/test_api has disappeared.
It looks like you can do uploads like this:
curl -X POST -F @.*** -F etc.
https://mushroomobserver.org/api2/images
Let me know if that doesn't work.
I tried your test URL verbatim (replacing xxx with my api key) and it
worked just fine. Can you give me your user id, and tell me
approximately
when you tried it, and I'll look it up in the log to see if I can figure
out what went awry.
Did I miss anything?
-Jason
On Thu, Aug 17, 2023 at 1:54 AM fran-aubry @.***> wrote:

It looks like the documentation for adding an observation is
incomplete.
Could you please help me figure out the proper format of the URL?
Namely:

The example url has a parameter named "name" while the online form
https://mushroomobserver.org/observations/new request a "scientific
name". Therefore, it is not obvious how each of the fields in the form
should be named in the url. Could you provide a complete mapping?
2.
For fields like "Used references" it is unclear what should be sent.
Both a string and a boolean if checked? Just a string?
3.
The readme refers to a file script/test_api to see how images should
be encoded. However it seems that this file doesn't exist.
4.
I've made a few requests using my key in this url:

https://mushroomobserver.org/api2/observations?api_key=xxx&name=Agaricus&location=Pasadena,+California,+USA&date=2016-08-06&notes=growing+in+lawn

I got a 200 on each of them. However I don't see them when I go to the
my
observations page.
Thank you in advance for you help.
Best regards.
—
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/issues/1558,
or
unsubscribe
<

https://github.com/notifications/unsubscribe-auth/AAYTNNMIKEPNU6EBVIDCL73XVWWZDANCNFSM6AAAAAA3TRHENE>

.
You are receiving this because you are subscribed to this
thread.Message
ID: @.***>

—
Reply to this email directly, view it on GitHub
<
https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1681714192>,

or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AA2CLA6YXYM6OVQR6EVT4MDXVW4UHANCNFSM6AAAAAA3TRHENE>

.
You are receiving this because you authored the thread.Message ID:
@.***>

—
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1683270594,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAYTNNLN3ZRPF424AFX2Q2TXV3LZRANCNFSM6AAAAAA3TRHENE
.
You are receiving this because you commented.Message ID:
@.***>

Thanks for the help. I manage to fix the observation creation process. I'm looking into image uploads now but I keep getting a 500 error. This is what I'm doing:
curl -X POST -F api_key='...' -F upload_url='https://media.istockphoto.com/id/1460769162/photo/two-mushrooms-in-a-dark-forest.jpg?s=2048x2048&w=is&k=20&c=Z8kbfYrgfGd4rUdkCVg9pt-d6_Tn2dF0VBSUuHhQ21I=' 'https://mushroomobserver.org/api2/images'
I suspect it is because some other parameters might be required?
I see the list here but it doesn't specify which ones are required:
Usage: copyright_holder: string (limit=100 chars); date: date (when photo taken); license: license; md5sum: string; notes: string; observations: observation list (must have edit permission); original_name: string (limit=120 chars, original file name or other private identifier); projects: project list (must be member); upload: upload; upload_file: string; upload_url: string; vote: enum (limit=1|2|3|4)

Could you take a look to see what I'm doing wrong with my request?
Thank you in advance :pray:

Great news that you've gotten your end working!  Now I gotta get our end
working. :)  Weird, though, because I've used this and our mobile app uses
this, and it works fine for us.  And in fact it is actually uploading the
image.  See:
https://mushroomobserver.org/images/1591185
https://mushroomobserver.org/images/1591186
https://mushroomobserver.org/images/1591187
https://mushroomobserver.org/images/1591188
It should be returning the image id in the result, but for some reason the
ids are nil which is causing an internal error.  I'm going to have to look
into this more.  I don't think it has anything to do with you.  Sorry about
that!  I'll get back to you tomorrow.
-Jason
On Mon, Aug 21, 2023 at 12:43 AM fran-aubry @.***>
wrote:

Thanks for the help. I manage to fix the observation creation process. I'm
looking into image uploads now but I keep getting a 500 error. This is what
I'm doing:
curl -X POST -F api_key='...' -F upload_url='
https://media.istockphoto.com/id/1460769162/photo/two-mushrooms-in-a-dark-forest.jpg?s=2048x2048&w=is&k=20&c=Z8kbfYrgfGd4rUdkCVg9pt-d6_Tn2dF0VBSUuHhQ21I='
'https://mushroomobserver.org/api2/images'
I suspect it is because some other parameters might be required?
I see the list here but it doesn't specify which ones are required:
Usage: copyright_holder: string (limit=100 chars); date: date (when photo taken); license: license; md5sum: string; notes: string; observations: observation list (must have edit permission); original_name: string (limit=120 chars, original file name or other private identifier); projects: project list (must be member); upload: upload; upload_file: string; upload_url: string; vote: enum (limit=1|2|3|4)
Could you take a look to see what I'm doing wrong with my request?
Thank you in advance 🙏
—
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1685631875,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAYTNNPQSGS3AJFYDP4SCALXWLRNNANCNFSM6AAAAAA3TRHENE
.
You are receiving this because you commented.Message ID:
@.***>

Thank you for looking into it.
Do you have an idea of when this will be fixed?

All is working now. URLs would have been more convenient but it's good enough for now.
Thank you for your time :pray:

Oh good.  I dove in deeper, and it should actually work.  And in fact we do
even have unit tests for it.  But the error messages are being swallowed
and hidden.
On Fri, Aug 25, 2023 at 6:20 AM fran-aubry @.***> wrote:

All is working now. URLs would have been more convenient but it's good
enough for now.
Thank you for your time 🙏
—
Reply to this email directly, view it on GitHub
https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1693134146,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAYTNNL44H6DKFHYGHMVAD3XXB36LANCNFSM6AAAAAA3TRHENE
.
You are receiving this because you commented.Message ID:
@.***>


New version: HiroshibaKazuyuki.VOICEVOX version 0.16.0

Pull request has been created with komac v2.0.2 :rocket:
Microsoft Reviewers: Open in CodeFlow

Automatic Validation ended with:

No errors to post.

(Automated response - build 684.)

@wingetbot waivers Add Validation-Executable-Error


feat: extract web dev utils

Description
Extracts the random web dev utilities to the https://hex.pm/packages/web_dev_utils package, so they might be utilized by other projects.

Current dependencies on/for this PR:

main

PR #31   👈

This comment was auto-generated by Graphite.


Tts accents not working on fr_FR locale

Hello,
I currently have an issue with locale fr_FR and the return from OWM :
confidence 0.711
[20:12:07] [Dialogue] was ask to end session with id 3023a423-ebac-45c5-a675-7a3
6c719f41a by saying 'La tempÃ©rature sera de 26 degrÃ©s  Ã  Vidauban'
[20:12:07] [Tts] was asked to say "La tempÃ©rature sera de 26 degrÃ©s  Ã  hidden"
every accent is replaced with Ã© and Tts says "A copyright tout droit réservé" instead of "é"
/var/lib/snips/skills/snips-skill-owm $ cat config.ini
[secret]
api_key=hidden
locale=fr_FR
default_location=hidden
/var/lib/snips/skills/snips-skill-owm $ locale
LANG=fr_FR.UTF-8
LANGUAGE=fr_FR.UTF-8
LC_CTYPE="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_PAPER="fr_FR.UTF-8"
LC_NAME="fr_FR.UTF-8"
LC_ADDRESS="fr_FR.UTF-8"
LC_TELEPHONE="fr_FR.UTF-8"
LC_MEASUREMENT="fr_FR.UTF-8"
LC_IDENTIFICATION="fr_FR.UTF-8"
LC_ALL=fr_FR.UTF-8
I'm using https://console.snips.ai/store/fr/skill_gy9rrnbBy2A7 with snips-skill-owm
Another exemple :
[20:20:28] [Asr] captured text "quel temps fait il à nice" in 3.0s
[20:20:28] [Asr] was asked to stop listening on site default
[20:20:28] [AudioServer] was asked to play a wav of 93.1 kB with id '6a355526-3604-45e1-94a5-35c55c89ffd' on site default
[20:20:29] [AudioServer] finished playing wav with id '6a355526-3604-45e1-94a5-d35c55c89ffd'
[20:20:29] [Nlu] was asked to parse input "quel temps fait il à nice"
[20:20:29] [Nlu] detected intent searchWeatherForecast with confidence score 1.000 for input "quel temps fait il à nice"
Slots ->
forecast_locality -> Nice (confidence: 0.955)
[20:20:29] [Dialogue] New intent detected searchWeatherForecast with confidence 1.000
Slots ->
forecast_locality -> Nice (confidence: 0.955)
[20:20:29] [Dialogue] was ask to end session with id 37d4b5fd-e7ce-4508-8fcf-c9a9027eff97 by saying 'le temps sera ensoleillÃ© Ã  Nice '
[20:20:29] [Tts] was asked to say "le temps sera ensoleillÃ© Ã  Nice"
Thank you for your help

My bad i didn't see the PR : https://github.com/snipsco/snips-skill-owm/pull/30