3 channel image (RGB) example

Hi jbms,
Do you have a working example for 3 channel image on Neuroglancer?  I am trying to troubleshoot if chunking on CloudVolume is failing or the shader control needs to be configured differently.  Perhaps, three channel image need to be converted to a gray scale image and then chunked?
Thank you so much for your help.
-m
#uicontrol vec3 color color(default="blue")
#uicontrol float min slider(default=0, min=0, max=1, step=0.01)
#uicontrol float max slider(default=1, min=0, max=1, step=0.01)
#uicontrol float brightness slider(default=0, min=-1, max=1, step=0.1)
#uicontrol float contrast slider(default=0, min=-3, max=3, step=0.1)

float scale(float x) {
  return (x - min) / (max - min);
}

void main() {
  emitRGB(
    color * vec3(
      scale(
        toNormalized(getDataValue()))
       + brightness) * exp(contrast)
  );
}

Please pardon my ignorance.
In the shader, you had
vec3(
      scale(
        toNormalized(getDataValue()))
       + brightness)

I thought normally, to initialise a vec3, one would need to provide three components: x,y,z. (?)

getDataValue() only returns the value of the first channel.  Instead you can use getDataValue(0), getDataValue(1), getDataValue(2).
Alternatively you can use an invlerp UI control rather than separate min/max sliders:
#uicontrol invlerp red(channel=0)
#uicontrol invlerp green(channel=1)
#uicontrol invlerp blue(channel=2)

void main() {
  emitRGB(vec3(red(), green(), blue()));
}

Thank you @xgui3783 and @jbms for the insights.  I tried both approaches in the shader.  However, they don't seem to show the image as expected.  I also made sure to include the RGB channel in numpy array on the CloudVolume side.  In fact, I didn't include these values in the first place so I thought that could also be an issue.  Could it be something I'm forgetting on the info file?  A status view seems to send tiles over the network so not sure where I'm failing.
Config from a Screenshot1
#uicontrol invlerp red(channel=0)
#uicontrol invlerp green(channel=1)
#uicontrol invlerp blue(channel=2)

void main() {
  emitRGB(vec3(red(), green(), blue()));
}

Config from a Screenshot2
#uicontrol vec3 color color(default="white")
#uicontrol float min slider(default=0, min=0, max=1, step=0.01)
#uicontrol float max slider(default=1, min=0, max=1, step=0.01)
#uicontrol float brightness slider(default=0, min=-1, max=1, step=0.1)
#uicontrol float contrast slider(default=0, min=-3, max=3, step=0.1)

float scale(float x) {
  return (x - min) / (max - min);
}

void main() {
  emitRGB(
    color * vec3(scale(toNormalized(getDataValue(0))) + brightness, scale(toNormalized(getDataValue(1))) + brightness, scale(toNormalized(getDataValue(2))) + brightness) * exp(contrast)
  );
}

Screenshot1

Screenshot2

{
  "data_type": "uint8",
  "num_channels": 3,
  "scales": [
    {
      "chunk_sizes": [
        [
          613,
          2753,
          1
        ]
      ],
      "encoding": "raw",
      "key": "1613_1613_10000",
      "resolution": [
        1613,
        1613,
        10000
      ],
      "size": [
        14712,
        11012,
        35
      ],
      "voxel_offset": [
        0,
        0,
        0
      ]
    }
  ],
  "type": "image"
}

The CDF plots shown in the invlerp controls seem to indicate that all data values are equal to 0.  You can also confirm that by open the selection details panel (control+rightclick) and hovering over positions with mouse.

I probably need to get familiarized with invlerp controls because changing the range or switching the arrows don't seem to show the image but filled RGB colors instead.  In the meantime, I see both zero and non-zero values on the panel from hovering over on the x-y view.  Could it be the values I'm providing with invlerp need to be adjusted?

Hmm, that's odd.  If you can share a sample volume that reproduces the issue I can take a look.

One doubt I have is if I processed numpy array properly on the CloudVolume.  I believe reshape should have assigned RGB channel as opposed to my initial try excluded this value.
Sure I can share a sample volume.  Where could I email you @jbms to my shared google drive?
      (width, height) = image.size
      #array = np.array(list(image.getdata()), dtype=np.uint16, order='F')
      array = np.array(list(image.getdata()), dtype=np.uint8, order='F')

      array = array.reshape((3, 1, height, width)).T
      # vol[:, :, z] = array
      vol[:, :, z, :] = array

My email is<EMAIL_ADDRESS>

It turns out the problem is that the chunk size you are using, [613, 2753, 1] is too large, as it results in Neuroglancer attempting to create a 3-d texture of size [613, 2753, 3].
The developer console shows:
texture_access.ts:317 WebGL: INVALID_VALUE: texImage3D: width, height or depth out of range
On my machine, webglreport.com (under webGL2 tab) shows a maximum 3d texture size of 2048, meaning all dimensions must be <= 2048.
Currently Neuroglancer always stores each chunk as a single texture.  In principle Neuroglancer could do rechunking client side to avoid exceeding the maximum chunk size, but that is not implemented.
If you rewrite your volume to use a smaller chunk size it should work.  I would recommend e.g. [512, 512, 1].

I see, the chunk size need to be smaller than what I had.  That is great to know and I should've looked at the Dev Console.  Thank you for reminding me.
I rewrote my volume with [512,512,1] chunk size and I think I came one step closer.  Although, it seems that images are flipped from the center on both vertically and horizontally.  And the Red and Green values seems to stand out since this image is mostly blue.  Could it be configurable on shaders end?  I also uploaded raw images (tiff) for you to view the originals @jbms Please advise me if you have any thoughts.  Thank you!

I'm not seeing your updated export, but it looks like you may have assumed the wrong data layout when writing your images.  It looks like you may have the channel dimension (rgb) incorrectly interleaved in your spatial dimensions.

@jbms Does Neuroglancer assumes RGB channel to be ordered RGB or BGR?

Neuroglancer doesn't make any assumptions about that.  You control that with the shader.
For example, with the sample shader I provided:
#uicontrol invlerp red(channel=0)
#uicontrol invlerp green(channel=1)
#uicontrol invlerp blue(channel=2)

void main() {
  emitRGB(vec3(red(), green(), blue()));
}

You can swap the channel= values to change the ordering.   Note that you can also change the channel using the UI control itself., by clicking on the "0", "1", "2" shown next to the CDFs.

It does appear to be channel dimensions are not shaped properly.  With grayscale images, conversion worked fine with X,Y,Z order.
With RGB tuples, I probably need to reshape it properly.  @jbms Does Neuroglancer expect precomputed data to be X,Y,Z,RGB or X,Y,Z,R,G,B or X,Y,RGB,Z or X,Y,R,G,B,Z ?

When using the "raw" encoding for the neuroglancer precomputed format, each individual chunk should be stored in lexicographical order c z y x.
So for example, if you have num_channels=3 and xyz chunk_size=[2, 2, 2], the successive values would correspond to:
Offset  0: c=0 z=0 y=0 x=0
Offset  1: c=0 z=0 y=0 x=1
Offset  2: c=0 z=0 y=1 x=0
Offset  3: c=0 z=0 y=1 x=1
Offset  4: c=0 z=1 y=0 x=0
Offset  5: c=0 z=1 y=0 x=1
Offset  6: c=0 z=1 y=1 x=0
Offset  7: c=0 z=1 y=1 x=1
Offset  8: c=1 z=0 y=0 x=0
Offset  9: c=1 z=0 y=0 x=1
Offset 10: c=1 z=0 y=1 x=0
Offset 11: c=1 z=0 y=1 x=1
Offset 12: c=1 z=1 y=0 x=0
Offset 13: c=1 z=1 y=0 x=1
Offset 14: c=1 z=1 y=1 x=0
Offset 15: c=1 z=1 y=1 x=1
Offset 16: c=2 z=0 y=0 x=0
Offset 17: c=2 z=0 y=0 x=1
Offset 18: c=2 z=0 y=1 x=0
Offset 19: c=2 z=0 y=1 x=1
Offset 20: c=2 z=1 y=0 x=0
Offset 21: c=2 z=1 y=0 x=1
Offset 22: c=2 z=1 y=1 x=0
Offset 23: c=2 z=1 y=1 x=1

However, if you are using cloudvolume to write, it should be taking care of that for you.

Yes, I am using CloudVolume.  And simply put, I am transposing (c, z, y, x) to (x, y, z, c) in numpy array similar to what has been working for grayscale images.  For grayscale images, it is (z,y,x) to (x,y,z).  But for whatever the reason, I'm only seeing a portion of the entire image as you see in attached images.  In this example, changing blue channel will only give me 1/3 (?) of the entire image.  Not quite sure where I'm failing at this point.

After all, I was able to get the working example for structuring the numpy array in CloudVolume for RGB channel images.  Thanks for the help here also.


⚠ ku-s has degraded performance

In e3be214, ku-s ($KU_API_STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 15110 ms

Resolved: ku-s performance has improved in 8852900.


^ is an annoyance

Given for instance this typings.plotlyDotJsLib.plotlyDotJsMod.^.newPlot('myDiv', data, layout)
What we probably want is to work with plotlyDotJsMod and that it contains everything that is needed. As it stands right now you may have to reference things in both there and in its ^ object.
Most people rename ^ to something else now, we could for instance use a capitalized version of the library name instead. What I do consider a plus with ^ is that as soon as you know the pattern it's predictable and easy to know where to look.
Another thing we could do is to output object instead of package, but that has some other negative consequences.

Always outputting object instead of package for one ruins our ability to generate companion objects with apply methods (because they cannot be nested in objects extending js.Object). Additionally, tooling is way worse for scala files spanning thousands of lines which motivated the move to packages in the first place.
One thing we could investigate would be to split values and types, so interfaces, type aliases and scala objects go in one hierarchy with packages, while all javascript values (class, val, def) go into objects.
This would resolve the need for ^, let us generate companion objects, keep tooling happy.
On the flip side it would come with an extra import tax, unless we come up with a particularly clever encoding. I'll think about this for a while

I'm publishing an updated encoding now which fixes most of this problem.
All top-level val and defs are now converted to top-level objects. We're still keeping (most) ^ objects, because they are a reference to the module itself, and you might need to refer it for instance in these two cases:

To use classes which are exported as a module in commonjs (defined in Typescript as export = class Foo {}, used in Scala as new typings.node.eventsMod.^())
If you want to mutate top-level members (typings.std.^.onerror = (x, _, _, _, _) => typings.std.console.warn(x))


Added oauth2 adapter for meetup.com/api

...and included in example app

cool, thanks @softprops!


Enable configuration of ImageJob timeout

The timeout property is not currently configurable through the top-level API.
Basically I did what is outlined by this comment (also tangentially related to https://github.com/openseadragon/openseadragon/issues/880, as I had problem  similar to what is outlined there).
I am not sure where tests should go for this fix. I added a new file called configuration.js with some TODOs, but there may be a place that is better suited.

@markmatney Looks great! Thank you for doing this.
For the tests, how about you change the test filename to imageloader.js? We should probably have more tests for that module, so it's good to start a space for it now.

Alright, ready for review.

Beautiful! Thank you for putting this together so nicely :)


Further simplify idx error message parsing

The simplifications from #35 were great, but it can be taken further. Construct and use the error parsing regexs directly.

Thank you for always taking improvements to the next level. 👍


Split heavy calculations from Rmd into a separate script

The Rmarkdown template (src/report.Rmd) contains a section "calculate" which loads CSV files, computes summary dataframes, writes some additional TSV files, and saves its state as an .Rdata file, but does not render plots or user-facing tables on its own. The visual elements are in separate, downstream sections in the Rmd template.
Rmarkdown runs these calculations end-to-end for each output report format (HTML and PDF). To avoid this redundant work and reduce overall runtime by a bit, and for separation of concerns and the convenience of downstream consumers, let's:

Move these calculations to a separate R script that the workflow will run before create_report.R.
Change the scripts to take each file input explicitly, rather than a shared path prefix.

Addressed in PR #14


Add RHEL Conversion VM kickstart template

Adding development version of RHEL-based kickstart file.
Downstream repos will be added before building the VM Image.

@nyoxi Hi, I added RHEL version of kickstart script which should serve as a starting point for OpenStack release delivery.


Update dependencies to latest stable versions

This PR updates ken's dependencies as revealed through lein ancient (sans org.clojure/clojure):
[manifold "0.2.4"] is available but we use "0.2.3"
[org.clojure/tools.namespace "1.3.0"] is available but we use "1.2.0"

Release version 1.0.2 will follow this upon merge.

@greglook I'm going to append this PR after I tag amperity/ken with a new release version so that it picks up the changes from that PR, so I'll request re-review once I'm done with that.


Add additional hardening to pcs-sap-hardening

In response to customer requests we added additional hardening rules to pcs-sap-hardening. The changes needed are:
1: Replace existing rule by rule from pcs-hardening profile    with direct use of the pcs-hardening-sap profile.
2: Add new loop to pull in additional rules from the cis_server_l1    profile. The rule list is defined by RULES_FROM_CIS variable
3: Add comments to explain why certain rules were not included.
The new SAP hardening results in a CIS Level 1 pass rate of 81.2% compared to 66.4% for the existing SAP hardened image.

Image was generated from these changes in:
https://build.suse.de/package/show/home:esampson:keg-images:SLE-15-SP4/SLES15-SP4-SAP-Hardened-BYOS
QA results can be found at: https://jira.suse.com/browse/TEAM-8046

I am not a fan of all the comments for rules we are skipping. IMHO we should track that elsewhere.

@jgleissner have changed PR to target develop branch.
@rjschwei have removed lines from script and put same info in the commit message. We do track the same information in confluence pages and jira's but I don't think those locations are visible outside of SUSE.

@brett060102 jira and confluence are not visible outside that is correct. Our documentation for the images points to the upstream profile. If we need to make information public about what rules did not get applied and why can we put that information there?

@rjschwei Thank you. The change to the profile upstream is in process.


Read config kafka

Fixes #

Proposed Changes

Read config-kafka configmap and get the broker address from there
Needed to make a func in eventing-kafka public and use it for parsing the configmap data: https://github.com/knative-sandbox/eventing-kafka/pull/847
Upgraded eventing-kafka dep to main branch. This also affected the eventing dependency version too.

Release Note

Docs

cc @devguyio @matzew

/test pull-knative-sandbox-eventing-kafka-broker-integration-tests
Failed to clean the resource "br" : timed out waiting for the condition

/lgtm
/assign @devguyio

/approve

/test pull-knative-sandbox-eventing-kafka-broker-integration-tests

Hi Ali, I want to share what's happening here.
Basically, the update to the eventing dependencies brings the new test Matthias wrote, see https://github.com/knative-sandbox/eventing-kafka-broker/pull/1194/files#diff-64634a925a6f4c9df07657597f4f3e08dca34517b71b2c42c7eb779406f05880R74.
However, the eventing webhook update is part of the nightly job update in https://github.com/knative-sandbox/eventing-kafka-broker/pull/1195.
Now #1195 is in, so we should be able to have a green run. (Prow rebases the PR automatically)
/retest

Different issue https://github.com/knative/eventing/issues/5700

test_runner.go:163: Couldn't initialize clients: unable to find available namespace

/retest

/retest

thanks @pierDipi !


no detection from Harmony app with Ha-bridge installed on MacOS

Hi
I have what appears to be ha-bridge fully up and running on a MacOS Mojave.
I successfully had a device included and tested.
However I can't get to have my Harmony app detecting a Hue bridge.
what am I missing please?
I spent a few hours investigating but I am desperate now.
here are my logs:
2018-12-17 21:31:57,676 [main] INFO  com.bwssystems.HABridge.HABridge - HA Bridge startup sequence...
2018-12-17 21:31:57,688 [main] INFO  com.bwssystems.HABridge.BridgeSettings - reading from config file: data/habridge.config
2018-12-17 21:31:57,973 [main] INFO  com.bwssystems.HABridge.HABridge - HA Bridge (v5.2.2RC2) initializing....
2018-12-17 21:31:57,980 [main] INFO  spark.staticfiles.StaticFilesConfiguration - StaticResourceHandler configured with folder = /public
2018-12-17 21:31:57,986 [main] INFO  com.bwssystems.HABridge.SystemControl - System control service started....
2018-12-17 21:31:58.002:INFO::Thread-0: Logging initialized @549ms to org.eclipse.jetty.util.log.StdErrLog
2018-12-17 21:31:58,003 [main] INFO  com.bwssystems.HABridge.util.UDPDatagramSender - Initializing UDP response Socket...
2018-12-17 21:31:58,006 [main] INFO  com.bwssystems.HABridge.util.UDPDatagramSender - UDP response Socket initialized to: 56258
2018-12-17 21:31:58,007 [main] INFO  com.bwssystems.HABridge.plugins.http.HTTPHome - HTTP Home created.
2018-12-17 21:31:58,009 [main] INFO  com.bwssystems.HABridge.plugins.harmony.HarmonyHome - Harmony Home created.
2018-12-17 21:31:58,034 [Thread-0] INFO  spark.embeddedserver.jetty.EmbeddedJettyServer - == Spark has ignited ...
2018-12-17 21:31:58,035 [Thread-0] INFO  spark.embeddedserver.jetty.EmbeddedJettyServer - >> Listening on <IP_ADDRESS>:80
2018-12-17 21:31:58.037:INFO:oejs.Server:Thread-0: jetty-9.4.z-SNAPSHOT, build timestamp: 2017-11-21T22:27:37+01:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
2018-12-17 21:31:58.054:INFO:oejs.session:Thread-0: DefaultSessionIdManager workerName=node0
2018-12-17 21:31:58.054:INFO:oejs.session:Thread-0: No SessionScavenger set, using defaults
2018-12-17 21:31:58.056:INFO:oejs.session:Thread-0: Scavenging every 660000ms
2018-12-17 21:31:58.084:INFO:oejs.AbstractConnector:Thread-0: Started ServerConnector@40ba5627{HTTP/1.1,[http/1.1]}{<IP_ADDRESS>:80}
2018-12-17 21:31:58.084:INFO:oejs.Server:Thread-0: Started @632ms
2018-12-17 21:31:58,259 [main] INFO  com.bwssystems.HABridge.plugins.harmony.HarmonyServer - setup initiated ....
déc. 17, 2018 9:31:58 PM com.fasterxml.jackson.databind.ext.Java7Support <clinit>
WARNING: Unable to load JDK7 types (annotations, java.nio.file.Path): no Java7 support added
2018-12-17 21:31:59,620 [main] INFO  com.bwssystems.HABridge.plugins.harmony.HarmonyServer - activity changed: [37131576] SHIELD TV
2018-12-17 21:31:59,623 [main] INFO  com.bwssystems.HABridge.plugins.NestBridge.NestHome - Nest Home created. No Nest configured.
2018-12-17 21:31:59,624 [main] INFO  com.bwssystems.HABridge.plugins.hue.HueHome - Hue passthru Home created. No Hue passtrhu systems configured.
2018-12-17 21:31:59,625 [main] INFO  com.bwssystems.HABridge.plugins.hal.HalHome - HAL Home created. No HAL devices configured.
2018-12-17 21:31:59,627 [main] INFO  com.bwssystems.HABridge.plugins.mqtt.MQTTHome - MQTT Home created. No MQTT Clients configured.
2018-12-17 21:31:59,628 [main] INFO  com.bwssystems.HABridge.plugins.hass.HassHome - HomeAssistant Home created. No HomeAssistants configured.
2018-12-17 21:31:59,629 [main] INFO  com.bwssystems.HABridge.plugins.homewizard.HomeWizardHome - HomeWizard Home created. No HomeWizard gateways configured.
2018-12-17 21:31:59,629 [main] INFO  com.bwssystems.HABridge.plugins.exec.CommandHome - Command Home for system program execution created.
2018-12-17 21:31:59,630 [main] INFO  com.bwssystems.HABridge.plugins.tcp.TCPHome - TCP Home created.
2018-12-17 21:31:59,631 [main] INFO  com.bwssystems.HABridge.plugins.udp.UDPHome - UDP Home created.
2018-12-17 21:31:59,632 [main] INFO  com.bwssystems.HABridge.plugins.vera.VeraHome - Vera Home created. No Veras configured.
2018-12-17 21:31:59,633 [main] INFO  com.bwssystems.HABridge.plugins.fibaro.FibaroHome - Fibaro Home created. No Fibaros configured.
2018-12-17 21:31:59,634 [main] INFO  com.bwssystems.HABridge.plugins.domoticz.DomoticzHome - Domoticz Home created. No Domoticz devices configured.
2018-12-17 21:31:59,635 [main] INFO  com.bwssystems.HABridge.plugins.somfy.SomfyHome - Somfy Home created. No Somfys configured.
2018-12-17 21:31:59,637 [main] INFO  com.bwssystems.HABridge.plugins.lifx.LifxHome - LifxDevice Home created. No LifxDevices configured.
2018-12-17 21:31:59,638 [main] INFO  com.bwssystems.HABridge.plugins.openhab.OpenHABHome - OpenHAB Home created. No OpenHABs configured.
2018-12-17 21:31:59,639 [main] INFO  com.bwssystems.HABridge.plugins.fhem.FHEMHome - FHEM Home created. No FHEMs configured.
2018-12-17 21:31:59,642 [main] INFO  com.bwssystems.HABridge.plugins.broadlink.BroadlinkHome - Broadlink Home created. No Broadlinks configured.
2018-12-17 21:31:59,651 [main] WARN  com.bwssystems.HABridge.dao.GroupRepository - Error reading the file: data/group.db - Does not exist or is not readable. continuing...
2018-12-17 21:31:59,651 [main] INFO  com.bwssystems.HABridge.devicemanagmeent.DeviceResource - HABridge device management service started....
2018-12-17 21:31:59,668 [main] INFO  com.bwssystems.HABridge.hue.HueMulator - Hue emulator service started....
2018-12-17 21:31:59,681 [main] INFO  com.bwssystems.HABridge.upnp.UpnpSettingsResource - Description xml service started....
2018-12-17 21:31:59,687 [main] INFO  com.bwssystems.HABridge.upnp.UpnpListener - UPNP Discovery Listener starting....
2018-12-17 21:31:59,688 [main] INFO  com.bwssystems.HABridge.upnp.UpnpListener - UPNP Discovery Listener running and ready....

Thanks

As this is similar to an echo finding the ha-bridge, please review your settings and turn on trace upnp to see if the hub can find the ha-bridge.

Hi @bwssytems
thanks for the tip
In parallel I successfully installed it on a temperary raspberry, so that I know what to expect
I spent a few hours trying to get it working on macos but have to give up now
I wouldn't be surprised that it comes from java security but I have no idea how to overcome it
thanks for the software, very good discovery
I will get another raspberry and integrate it into my home automation system


#trivial Fix #226

          1 Warning

:warning:
Any source code changes should have an entry in CHANGELOG.md or have #trivial in their title.

  Generated by :no_entry_sign: Danger

🚫 CI failed with log

Unfortunately using the ttl cache incurs some measurable performance costs, that would be proportionally more severe for the memory cache. It's also less relevant (as the memory cache is already TTL = The process lifetime)


Demo workflow does not consistently run to completion

Due to ongoing development the demo keeps getting broken. If nothing else works in lcr-modules at least the demo needs to work consistently.

Are there any specific issues that consistently comes up (specific modules or rules)?

It seems like it's missing some inputs (e.g. 99-20595T.bam) since the sample_id is in the samples.tsv but it seems like you are using it to test specific modules only. I'm familiar with the original demo bam files, but I'm wondering which modules uses the igv.seg files.
would it be possible to get samples that can be used with all the modules to limit the number of samples in the samples.tsv?

Kostia's currently working on this. I think he has an updated version in a branch too.

Okay! I'll be available if he needs any help

I just pushed the version of demo workflow that I was able to complete in dry run and on cluster to the branch demo_repairs


add enable_script_security option to parser

since keepalived 1.3.0, add an option enable_script_security so that any scripts failing above won't be executed.

add enable_script_security to parser/lexer.go and parser/parser.go.y
add test

Thanks for your PR. I've just released v0.4.2. https://github.com/yuuki/gokc/releases/tag/v0.4.2


Add NARROWS

NARROWS doesn't have, unfortunately, any public installation I can link to. To see it in action people would have to clone the repo and install it. Thus, I'm not sure it should be in builtwithelm.co, but it was easy to add it, so here you go :-)

Merged and deployed. Thanks! @emanchado FYI your primaryUrl is not currently loading.

Thanks! Yes, bad timing :-( My hosting is currently having problems. I hope I'll be able to put it back online soon.


[REFACTOR] OTLP receiver: Don't call NormalizeLabel in NoUTF8EscapingWithSuffixes mode

Some basic simplification of the OTLP translation logic, where it doesn't call prometheustranslator.NormalizeLabel if the translation mode is NoUTF8EscapingWithSuffixes. Behaviour doesn't change, the motivation is simply readability/maintainability.
Prior to the PR, prometheustranslator.NormalizeLabel is a no-op if the translation mode is NoUTF8EscapingWithSuffixes, so it's just confusing to the reader (and the doc comment is also misleading, as it doesn't take this mode into account).

Is this going to be in the release v3.1.0? Because when I try to update prometheus to v3.1.0-rc.0 I got errors from a dependency which is using an older version.
# github.com/grafana/loki/v3/pkg/logql/log
../../../go/pkg/mod/github.com/grafana/loki/v3@v3.2.1/pkg/logql/log/pipeline.go:108:58: not enough arguments in call to prometheus.NormalizeLabel
        have (string)
        want (string, bool)
../../../go/pkg/mod/github.com/grafana/loki/v3@v3.2.1/pkg/logql/log/pipeline.go:232:58: not enough arguments in call to prometheus.NormalizeLabel
        have (string)
        want (string, bool)

I believe this PR will resolve the errors. Otherwise I have to go ahead and update loki and get a new version and update grafana. Then when we have a release with this PR in to the same again.

@itsmylife there are no plans for it currently, although maybe it would make sense in conjunction with other OTLP fixes. I'm actually surprised that Loki would call NormalizeLabel, as it's meant for use in the OTLP translator.

@aknuds1 here are the usages https://github.com/search?q=repo%3Agrafana%2Floki+"prometheus.NormalizeLabel"&type=code
@bboreham told me he could add this into the release. Do you think adding this to the release is a good idea?

Do you think adding this to the release is a good idea?

I think it can't hurt at least. It's just a refactoring, and merely cleans up the code. No behaviour changes.


show, enable, and disable bonus token for a VC

What would you like to be added:
Provide UI for operators to see, enable, and disable bonus tokens for a VC, in the Virtual Cluster page.
Why is this needed:
Better operation experience.
Without this feature, how does the current module work：
We only have REST API in 0.10.y.
Components that may involve changes:
web portal

@fanyangCS
I have offlines synced with @xwzheng1020. They are pleased to continue the contribution. What's your opinion?

@mzmssg @fanyangCS - add my vote for supporting @xwzheng1020 's team work on it.

hello, we will handle it.

@mzmssg @xwzheng1020. Great! Thanks!

@xwzheng1020 - would you like to target this item to March Release or later release?

@scarlett2018 March Release.

@xwzheng1020 @fanyangCS - we need a more clear definition about how VC bonus is used end 2 end before implement it. i.e. how jobs are preempted.

We rely on HiveD to exploit idle resource.


Update of a project overwrites existing Cache settings.

Rundeck Release = Rundeck 2.10.0-1
When updating the project configuration, via "Edit Configuration..."the following cache settings will be overwritten with the defaults of.
project.nodeCache.delay=30
project.nodeCache.enabled=false
We use the following configuration with Ansible, and when this happens it kicks off a Gather Facts job which takes 67 minutes to complete. During this time the inventory disappears, & jobs will fail with a no nodes found error.
project.nodeCache.delay=1209600
project.nodeCache.enabled=true
With the above configuration the inventory is cached for 14 days. If an update is needed before that time, we'll kick off a node refresh job at a time that will not impact existing jobs.
I can think of a few improvements.
1 ) Do not overwrite the existing configuration settings.
2 ) Definition of the default values, for project.nodeCache.delay and project.nodeCache.enabled
If an update is made to the node configuration via "Edit Nodes..."
Only update then inventory for a new resource, or changed resource.

From a quick look through the code I don't believe this is an issue with the Ansible plugin, but is an issue with Rundeck itself. Can I ask you to test with another inventory source and show if the node caching settings change then too?

Issue has gone stale. Closing. If you need further assistance then please post again.


adds formats registry considerations

related #3196

@handrews I just wanted to surface this one up for you in case you didn't see it :) Your feedback is always welcome.


New implementation of /run support

This mounts a /run tmpfs into the container, with the initial contents
copies from the /run in the base image, unless NoRunFs is set in the
HostConfig.

Additionally NoRunFs is always set during a docker build, which means
any setup of /run in a Dockerfile is saved in the image to be copied
into the final /run tmpfs when a container is started.

Docker-DCO-1.1-Signed-off-by: Alexander Larsson<EMAIL_ADDRESS>(github: alexlarsson)

Docker-DCO-1.1-Signed-off-by: Dan Walsh<EMAIL_ADDRESS>(github: rhatdan)
Docker-DCO-1.1-Signed-off-by: Dan Walsh<EMAIL_ADDRESS>(github: rhatdan)

Review session with @tiborvass @unclejack @crosbymichael
@rhatdan Can you please explain why --tmpfs /run (as implemented by #9586) isn't sufficient? It seems that populating the content of /run could be achieved through a container entrypoint script.

/run on TMPFS is a standard.  Is there a distribution where this is not true.
How do I see what is under the /run or the /tmp if something is mounted over them?
  We want to ship systemd based images, which expects /run to be on tmpfs by default as is the standard.
Lets not make running docker containers harder and more fragile by not accepting this patch.  At least allow us to customize the daemon to make this the default behaviour.   I see no reason to require /run to not be a tmpfs.

@rhatdan If you are using systemd images, shouldn't the setup scripts populate /run for the daemons that you are executing?

I want to allow users to do a Dockerfile that does the following
FROM rhel7
RUN yum install httpd; systemctl enable httpd
CMD /sbin/init
This will run the httpd service the way it was designed, no special hacking of init scripts, as simple as can be.  You just install the service the way you would on bare metal and use its service script to manage it.

In this case systemd will populate the tmpfs if you run with the --tmpfs /run, but will blow up if the user forgets the command.
Systemd-tmpfiles will take care of populating /run
In other cases people using --tmpfs /run will blow up apps since there is no way to prepopulate /run and certain images expect it.  Having /run as the default, or allowing users to configure it to be default, solves all problems.

I would actually like to know what having /run as tmpfs actually breaks, considering the patch copies the contents of /run off the image.

It would not break anything with the copy but I'm not a fan of the mount outside of the containers mount namespace and libcontainer is not the place to be copying files.
I don't think it's too much to ask a user to add --tmpfs /run in order to run an advanced piece of software like systemd in their container.  For httpd they are still going to have to map ports or the like.
Maybe we can make the argument that TMPFS /run should be allowed in the Dockerfile because it is portable and not machine specific.

WRT /var/run and read-only - for what it's worth, modern ostree will automatically make all of /var a tmpfs if / is readonly.  See https://git.gnome.org/browse/ostree/commit/?id=ff6883ca0655ac8844cd783caf6a7d8815515ba3
I think it's a pretty good default, as if you boot a read-only image, you wouldn't expect persistence, but it's nice if the system does boot out of the box.

I'm not disagreeing that /run should not be in tmpfs by default, the problem is that for docker's entire lifetime it's not and so changing that now is a little harder.  Copy is the solution to that problem but it comes at a cost of performance and complexity

httpd was just an example.  Any system service provided by the distribution would work.  Is there another way we could copy the files off of /run to preserve it.  Your new patch is going to have the same problem,  If we had a standard way of preserving the underlying image content on top of the tmpfs, we could solve the problem.

TMPFS in dockerfile has the same problem
FROM rhel7
TMPFS /run
RUN touch /run/dan
Would not work.

I have no problem making /run-tmpfs optional as a daemon option.  Which should satisfy the people who expect /run not to be a tmpfs, we would choose to ship our docker daemon with --runtmpfs enabled.

In your dockerfile example I would expect that to work because the user explicitly set to mount /run in tmpfs so it's their bug, not ours.
Do you think a daemon option would be better than a Dockerfile instruction?

@rhatdan did you want to get this into docker? it needs documentation, including some information telling users when and why they would choose (or not)

Sure.  I am waiting for @crosbymichael to comment on patch I sent to libcontainer, to handle copying directory contents from the image to the tmpfs file system that is being mounted over it.
https://github.com/docker/libcontainer/pull/331

@crosbymichael @rhatdan Patch for libcontainer wasn't accepted. Should we just close this too and find consensus first?

I fixed up the calls to libcontainer to remove any archive use from it.  Patch submitted for libcontainer.
https://github.com/docker/libcontainer/pull/495

@crosbymichael We now pass down the tar/untar commands to libcontainer from docker.

@LK4D4 Reworked to hopefully match @crosbymichael requirements.

@crosbymichael Can I have your attention here?

Can you please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "run" git@github.com:rhatdan/docker.git somewhere
$ cd somewhere
$ git rebase -i HEAD~4
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

This will update the existing PR, so you do not need to open a new one.

@rhatdan you forgot to sign of your last commit

Something seems a bit off to me.  In general I don't like to assume anything about what is running in the container.  The default mounts that docker sets up today are all /proc and /sys, IIRC, and those are essential to linux.  Are we saying that /run as tmpfs is just absolutely fundamental to the behavior of Linux?  I don't know @crosbymichael latest opinion, but --tmpfs /run seems a bit more reasonable to me.
As a side question, is something broken if /run isn't a tmpfs?  Is there a basic assumption somewhere that /run must be ephemeral?

On 04/20/2015 01:30 PM, Darren Shepherd wrote:

Something seems a bit off to me. In general I don't like to assume
anything about what is running in the container. The default mounts
that docker sets up today are all /proc and /sys, IIRC, and those are
essential to linux. Are we saying that /run as tmpfs is just
absolutely fundamental to the behavior of Linux? I don't know
@crosbymichael https://github.com/crosbymichael latest opinion, but
--tmpfs /run seems a bit more reasonable to me.
As a side question, is something broken if /run isn't a tmpfs? Is
there a basic assumption somewhere that /run must be ephemeral?
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/8478#issuecomment-94516426.

How come I don't see this on github?  The biggest problem we have with
/run not being on a tmpfs is that systemd will not run without
it on a tmpfs, actually it will but it will attempt to mount /run as a
tmpfs and will blow up.
Secondly, I want to get both /run and /tmp as tmpfs by default so that
it is simple to do a
docker run --read-only fedora /bin/sh
And most apps will run.  Apps expect to be able write to /tmp and /run.
/run is a tmpfs on every modern Linux Distribution and /tmp is also on
most.  Having these be the same in containers makes sense.

How come I don't see this on github?

Interesting question, I just trashed the email, luckily you noticed it

@rhatdan after reading over the thread, i deleted my comment, I didn't think it was valid.  I just struggle with how much we are assuming about the target env.  There's probably not a big harm in doing /run and /tmp but then I've seen other PR that talk about putting things in /etc and /var/run.  Where do we draw the line?  We draw this fine line between what systemd requires and what is "standard" in Linux because what major distro isn't influenced by systemd now.  Wasn't the notion of /run really introduced by systemd to begin with?  Not really important...
I feel the image should be able to specify this requirement and we shouldn't make this global.  So the TMPFS dockerfile command seem much more the right way to go.
If somebody used TMPFS /run then they are deciding to make /run ephemeral and then there is no need to prepopulate /run, right?  I'm in favor of making this a Dockerfile command and adding --tmpfs to docker run.  Then we could back out the change in libcontainer.

/var/run and /run are basically the same thing.  On modern systems /var/run is a symlink to /run.  I believe Ubuntu was the first to standardize on /run as a tmpfs and Fedora/systemd followed.
I actually like the idea of specifying in the base image or individual images where the TMPFS directories are.  So I have no problem moving to this model, but you still need to be able to store files/directories in the underlying directory and copying it into the TMPFS.
For example apache expects their to be a /run/httpd directory and will not work without it.  Putting this in the base image and allowing libcontainer/docker to preserve it onto the tmpfs is necessary.

Where do we draw the line?

I draw the line at matching what standard distros define.  /run and /tmp as tmpfs.  But if we go with putting it in the base images, I can go along with that.  The goal with /tmp is for --read-only, otherwise not that important.

We draw this fine line between what systemd requires and what is "standard" in Linux because what major distro isn't influenced by systemd now.

Not sure the answer to this question.  But trust me I am trying hard to get systemd to change some of its decisions.  Some times I feel like I am between a rock (systemd) and a hard place (docker).  :^)

Wasn't the notion of /run really introduced by systemd to begin with?

No I think it pre-existed systemd.

If somebody used TMPFS /run then they are deciding to make /run ephemeral and then there is no need to prepopulate /run, right? I'm in favor of making this a Dockerfile command and adding --tmpfs to docker run.

I would prefer a syntax like -v tmpfs:/tmp or -v /tmp:T

Then we could back out the change in libcontainer.

No we need the Premount/Postmount behaviour in libcontainer to be able to get the content off of the image into the tmpfs.
I could see use cases where I would want to run different containers with different /etc/httpd or /var
docker run --read-only -v tmpfs:/var httpd
Where httpd would be mounted as a tmpfs but the contents off of the image would be copied into it.

@rhatdan How does httpd work on a normal Linux distro if /run is empty.  Prepopulating a tmpfs seems a bit dirty to me because I can't really find an analogy to a typical linux distro.  If I create a tmpfs mount then the folder is empty and if I want to populate it I must write script that runs on startup, or use systemd-tmpfiles(?).
  I'd personally opt to not support prepopulating a tmpfs.
The red flag to me is that when I see the libcontainer changes the hacker in me gets excited because you just gave me the ability to run arbitrary commands through libcontainer.  This is a blocking hook that I could abuse like crazy (granted it's not exposed through Docker, but don't worry I'll get creative :) )  I prefer a bit more of a focused implementation in libcontainer to support prepopulating a tmpfs, but then I can't think of a clean design.
I think Docker should be as opinionated about user space as Linux is.  This means while every major distro supports /run tmpfs, it is still just a defacto standard among distros and init systems.  But Docker should support the ability to do such a thing because if you are running a Ubuntu or Fedora based image you would expect /run to be tmpfs.  This reinforces in my mind that the right approach is to support it in the image.
I'd prefer TMPFS/--tmpfs over -v.  The reason being that if it's a volume then the Dockerfile would have to be something like VOLUME tmpfs://:/run.  That sort of breaks the design of VOLUME in Dockerfiles because current VOLUME does not allow specifying the left side of : (well you're not supposed to).  We would have to make an exception for tmpfs:// which seems hacky.

How does httpd work on a normal Linux distro if /run is empty.

Because it has an init script that does the mkdir (or the equivalent configuration for systemd).
In Docker, the Dockerfile is as close to an "init script" as we have, save entrypoint workarounds.

Content in /run is populated via the init script or systemd-tmpfiles.
We have gone back and forth on the PreMount/PostMount command, this is what the upstream wanted so we gave it to them.  Previous patches had the "tar" happening on the libcontainer side.
I think having premount/postmount could be handy for other functions like notifications of mounts.  Not looking at this from a Malicious point of view.  I guess if you think of libcontainer as a priv separation point of view, where libcontianer would have more privs then the caller, the execution of these commands could be risky.  But I don't believe that is the way it is used currently.

I agree with @crosbymichael that we should look for a more general solution here. I see the benefit of allowing to mount tmpfs directories and I understand that systemd requires /run to be mounted that way.
I don't think making this specific to /run is the way to go, though. A more general solution like Mike proposes with --tmpfs allows to fulfill systemd's requirements and is not coupled to it at the same time. Besides, this is a hard linux implementation, using --tmpfs would allow us to abstract the implementation by OS. Windows is coming.
So, I'm :+1: on allowing to mount tmpfs directories and I'll be glad to push it forward with a general solution.

Well I am now thinking of moving the systemd requirements into a different patch set and adding a --systemd switch.  There are lots of requirements that systemd and docker do not agree with, and I don't see getting either upstream to agree.  Therefore I think we need docker run to support --systemd flag which would tell it that the container will run with a systemd as an init program and set up the container correctly.
Then container that do not want to run in systemd mode can continue to run as they currently do.  Only problem with doing this would be users confusion in running systemd based containers failing because they forget to run with the --systemd flag.

@rhatdan docker already knows how to detect that systemd is running via libcontainer, see https://github.com/docker/docker/blob/53bef64804c6dae6662a7d55c3bb3e48b3e5dfdf/daemon/execdriver/native/driver.go#L62 for instance.
If we add --tmpfs as a flag, there is nothing stopping us to set --tmpfs /run by default when systemd is running. That way we get both, a generic implementation and a systemd sane default.

It can tell if systemd is running on the host ,but not if it will be running as PID 1 in the container.  systemd expects things like the SIGTERM to have different meaning then what docker expects.  It expects to have /run and /sys/fs/cgroup mounted in the container,  I want it to be able to write journal data to the host OS.  There are a few other features that are also required to fully support systemd as pid1 in a container.

Implementation makes sense for me. But tests failing hard.
Also this doing this by default, so I moving to design-review.
ping @docker/core-maintainers

Just coming up to speed on this issue, but I'm not following something. Based on what I've read so far (in this PR and http://thread.gmane.org/gmane.linux.redhat.fedora.devel/146976) it appears that /run mounted as tmpfs is standard, right? If so, why wouldn't Docker do the same thing by default? Why would we want to require people to add a flag to enable something that would be enabled by default if they were running natively, outside of Docker?
Also, if we did end up with a flag, I'm not in favor of one on the daemon because if we think some people may not always want it then I'd prefer for it to be on a per-container basis. Or at least allow it for both so that a container can override what the daemon's default is.

After playing and shipping this patch for a while.  We are seeing other problems with it.  Biggest one being docker commit does not work the way one would expect.  docker commit only saves the underlying image, not anything mounted on top of the container image.  This means someone doing a
docker run -ti -n myhttpd image /bin/sh; yum install httpd; mkdir /run/httpd; ^d
docker commit myhttpd httpd
Would not end up getting /run/httpd by default.
I am now thinking the best way to handle this is just to have a big --systemd flag.
docker run --systemd ...
Which would set the container up in the mode that systemd would expect and would mount /run and /sys/fc/cgroup the way systemd would want, as well as generate journald content on the host and send the proper signals to systemd when users do a docker stop ID.
But for rank and file containers, we leave /run on the image.

We've been running systemd in Docker in production for a few months and it mostly works and has helped us do many things where multiple Dockers wouldn't have made sense. Big +1 on --systemd. We still need --tmpfs to be able to use --read-only more widely, without breaking apps that need small scratch directories where -v or VOLUME would be overkill (or having the scratch disk on physical disk would be bad for performance).

I agree I like the idea of --tmpfs although I think -v tmpfs:/PATH would be more consistent.  If docker upstream would decide which to do, I could have a patch available in a couple of hours.
I will push to get --systemd pull request together by next week.

But remember with --systemd or --tmpfs, we need to document that docker commit will not save any content that is stored on tmpfs, or for that matter any volume mounted content onto the new image, which might surprise some users.

Closing this and replacing it with
https://github.com/docker/docker/pull/13525


Add a dummy.go file to allow vendoring config

This is needed for go mod vendor to pull in Tekton's config directory.

/ok-to-test


Emulated device not displaying history

I'm trying to upload historical data to an emulated humidity sensor, but finding that I can't see the data in DT studio. Here's a reproducible example that creates a device, sends some timestamped events, then retrieves and lists the device's event history:
import os
import disruptive as dt
from datetime import datetime, timedelta, timezone
from dotenv import load_dotenv
from time import sleep
load_dotenv()

_KEY_ID = os.getenv('DT_SVC_KEY', '')
_SECRET = os.getenv('DT_SVC_SECRET', '')
_EMAIL = os.getenv('DT_SVC_EMAIL', '')
_PROJECT_ID = os.getenv('DT_PROJECT', '')

dt.default_auth = dt.Auth.service_account(_KEY_ID, _SECRET, _EMAIL)

vsensor = dt.Emulator.create_device(_PROJECT_ID, dt.Device.HUMIDITY, display_name="test_timestamp")

device_id = vsensor.device_id
print(device_id)

# publish some historic events
for i in range(5):
  timestamp = datetime.now(tz=timezone.utc) - timedelta(minutes=10-i) #(tz=timezone.utc)
  dt.Emulator.publish_event(device_id, _PROJECT_ID, data=dt.events.Humidity(celsius=20.0+i, humidity=50.0+i, timestamp=timestamp))

# wait...
sleep(5)

#now get events back
event_history = dt.EventHistory.list_events(device_id, _PROJECT_ID)

for e in event_history:
  print(e.event_type, str(e.data.timestamp))

which produces something like:
emuc3bdu268l9ck9sfrm9tg
networkStatus 2021-06-26 08:02:17.111216+00:00
networkStatus 2021-06-26 08:02:16.927892+00:00
networkStatus 2021-06-26 08:02:16.746447+00:00
networkStatus 2021-06-26 08:02:16.561380+00:00
networkStatus 2021-06-26 08:02:16.394273+00:00
humidity 2021-06-26 07:56:16.984021+00:00
humidity 2021-06-26 07:55:16.796853+00:00
humidity 2021-06-26 07:54:16.621677+00:00
humidity 2021-06-26 07:53:16.448767+00:00
humidity 2021-06-26 07:52:16.280573+00:00

note the networkStatus events (with different timestamps)
and what I see in DT studio is the latest value (as expected) in the top box, but the event history is empty, whatever I try, so I cant actually view the data I sent.
NB if I send current events, I don't get this problem
Update: I also get this behaviour using the emulator API directly

@virgesmith Like you suggested, this is a bug related to how the emulator generates the NetworkStatus events. All events published to the emulator (except NetworkStatus and Cloud Connector events) gets an associated NetworkStatus event. DT Studio uses these events to display when the sensor was offline. The bug here is basically that the timestamp you've specified in the Humidity event is not considered when generating the associated NetworkStatus event. It always uses the current timestamp.
I have a fix for this locally, but as the fix requires an update on our backend, it won't be out until Monday. If you want a temporary workaround for this, I would recommend sending a NetworkStatus event yourself for each Humidity event and making sure the timestamp is correct.
I'll update this issue when the fix is out.

wow, quick response, thanks! It's not urgent so I'm happy to wait

Apologies for the delay on this issue @virgesmith. I just rolled the fix out to production, so the accompanying NetworkStatus events have the same timestamp as the events you publish. This means the graph in DT Studio won't show as offline when you publish events in the past.
Note that this only applies to new events being published. The events that have already been published will still show as offline, unfortunately.

thanks @vegather all seems fine now


Does kmonad work on wlroot(wayland).

Will it work on wayland compositors built on wlroots.

It does @sadikeey. On Linux KMonad uses evdev to intercept, process, and emit keyboard events at the kernel level. I.e. it does not depend on anything but a Linux driver ubiquitously shipped with most distributions.
I hope I've answered your question sufficiently simply for you to understand.
I would further ask you to refrain to ask generic questions of a github project under issues, specifically when the README specifically outlines in its Introduction:

Additionally, if you need any help or just want to say hi, you can join our
Discord server or jump into our IRC channel
(#kmonad@irc.libera.chat), which is also bridged with matrix (#kmonad:libera.chat).

Close this issue.


enabling/disabling Steve restarts the Rancher container

What kind of request is this (question/bug/enhancement/feature request):
enhancement

Enabling/disabling Steve restarts the Rancher container, which causes Rancher UI is not available temporarily. The restart is expected because the steve feature is not dynamic.
We should add some kind of hints or notices in the UI for this behavior.
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):
latest master-head fa96c0c3d
Installation option (single install/HA):
single install

closed for the sake of https://github.com/rancher/rancher/issues/25421


OKAY, FOR THE INCLINED? YOU ARE STUDYING CIVIL ENGINEERING AT THE COMMUNITY COLLEGE OR STATE COLLEGE RIGHT?


Enforce max length of Title to 50 chars

Picked up https://allihoopa.com/s/aKPn8Z6e and tried to drop. The SDK cuts of the title so it only contains 50 unicode chars.

I have a commit in this PR from @erkkah, this happened because he pushed directly to master and skipped develop. I branched my feature branch of master.

This pull request removes the validation of the default title's maximum length (50 code points) and instead truncates the default title if needed. This doesn't change any API.


getMatchHistory return 403: forbidden

other methods work fine, just getMatchHistory returns 403:forbidden, here is the code:
LolApi.getMatchHistory 40200522, null, (e, matchlist) ->
if e?
console.log e
return

Same problem!!
Error getting match history: 403 Forbidden undefined
help pls

The match history endpoint was deprecated by Riot 3 months ago so that would explain the 403.
Instead of match history, matchlist and match are used for retrieving similar data.
Matchlist: https://developer.riotgames.com/api/methods#!/1053/3617
Match: https://developer.riotgames.com/api/methods#!/1058/3656
Heres the forum thread discussing the deprecation: https://developer.riotgames.com/api/methods#!/1053/3617
This library hasn't been updated yet so currently it doesn't support either of the new methods.
If you're looking for an alternative, I would suggest 'lolapi': https://github.com/emmorts/lolapi

Unfortunately I haven't been doing much with the LoL API, and haven't been maintaining it as well as I should have. I'll try and update this as soon as possible


Make sui-indexer writer work with tidb

Description
This PR makes some changes to allow sui indexer writer to work with tidb:

TiDB does not support UDFs and stored procedure (but Mysql does). For doing advance_partition, we invoke it directly through a query in a transaction
Remove udf and stored procedure from mysql migraitons
Some columns which store bcs are not big enough as blob types in mysql/tidb. Making them mediumblob instead

Test plan
Running locally and ingesting data into a tidb cluster

looks like you have been able to run indexer with TiDB already, can you write a quick note about how to run it with TiDB? any access do I need to get? thanks!


Removing doubled-up tiles

Hello!
As you can see in the example video below, at least three images are stacked on top of each other (note image to the right and image peeking out at the bottom) when I expect only one image to be present. Do you have any suggestions on how to remove the two back images? I can't seem to identify a parameter in either leaflet.LMap or leaflet.LTileLayer that will resolve this.
https://user-images.githubusercontent.com/44422832/218005796-c4b595f4-69ef-4c14-bc60-aed290b200e3.mov

We probably need some code to see what you are doing.

Did you print the layout HTML to make sure you did not create 3 LMap based on the way you defined your UI in Python?

I've seen no evidence of 3 or more LMap objects in the html, though I must admit I'm not experienced on the html side of things. Another thought is that somehow tiles at different zoom levels are being displayed simultaneously?
Here is a minimum reproducible example:
"""Image tile serving with large-image and trame."""

import io
import json

import large_image
from aiohttp import web
from trame.app import get_server
from trame.ui.vuetify import SinglePageWithDrawerLayout
from trame.widgets import leaflet, vuetify

server = get_server()
state, ctrl = server.state, server.controller

image = "/Users/robertcollar/Documents/CACB_Field_Work_2022/sample_photos/test/pyramids/IMG_1531.tiff"

# Open raster with large-image
class image_handler():
    def __init__(self, image):
        self.open_image(image)
    def open_image(self, image):
        self.src = large_image.open(
            image,
            encoding="PNG",
        )
    async def metadata(self, request):
        """REST endpoint to get image metadata."""
        return web.json_response(self.src.getMetadata())

    async def tile(self, request):
        """REST endpoint to server tiles from image in slippy maps standard."""
        z = int(request.match_info["z"])
        x = int(request.match_info["x"])
        y = int(request.match_info["y"])
        tile_binary = self.src.getTile(x, y, z)
        return web.Response(body=io.BytesIO(tile_binary), content_type="image/png")

handler= image_handler(image)

my_routes = [web.get("/metadata", handler.metadata), web.get("/tile/{z}/{x}/{y}.png", handler.tile)]

@server.controller.add("on_server_bind")
def app_available(wslink_server):
    """Add our custom REST endpoints to the trame server."""
    wslink_server.app.add_routes(my_routes)

# -----------------------------------------------------------------------------
# UI
# -----------------------------------------------------------------------------

with SinglePageWithDrawerLayout(server) as layout:
    layout.title.set_text("Large Image Trame Web App")

    with layout.toolbar as toolbar:
        toolbar.style = "z-index: 1000"
        vuetify.VSpacer()

        vuetify.VSwitch(
            hide_details=True,
            v_model=("$vuetify.theme.dark",),
        )
        vuetify.VProgressLinear(
            indeterminate=True,
            absolute=True,
            bottom=True,
            active=("trame__busy",),
        )

    with layout.content:
        with vuetify.VContainer(
            fluid=True,
            classes="pa-0 fill-height",
        ):
            with leaflet.LMap(zoom=("zoom", 1), maxZoom=15):
                # tiles
                image_viewer = leaflet.LTileLayer(url="/tile/{z}/{x}/{y}.png", noWrap=True, maxZoom=15)

if __name__ == "__main__":
    server.start(port=8080)

Indeed I don't see anything wrong at first glance...
Not sure if the issue is coming from the vue2-leaflet library we are exposing...

Got it – what would you suggest I do to diagnose this issue?

Not sure but since we would need a deeper investigation, we may have to rely on some funding source (which you may already have with Bane?)

Hi @jourdain, unfortunately there is no funding agreement between us! Simply an open source enthusiast trying to understand (and hopefully overcome) the barriers to building useful tools (in my case for a graduate research project!).

No worries and hopefully we'll manage to figure it out.

Hey team! Quick update — resolved the doubled-up tiles issue. To summarize, when an image cannot be broken up into equidimensional tiles, the edge style parameter — documented here – determines how the excess space beyond the image's edge is filled. Setting that parameter to "#DDDDDD" fills the excess space with the same default color as the trame-leaflet background (see example below)!


Renaming a QuickExt-ed COMP requires uncook-recook

Seems like dependency is not updated, as well as the extension still remaining in TD memory and somehow still calling from/to the non-existing comp???

hmm so an OP's .id attribute doesn't change between renames... 😃 but it does if you move it somewhere else
and since extensions reinit when a comp is moved, maybe I can just cache this id instead of the reference to the clearly non-existent ghost OP after a rename


Add front message feature

Feature
[Add] - Possibility to add a custom message in front of server embed
This feature can be useful to create for example categories
Exemple

[
    {
        "type": "SourceQuery",
        "game": "Insurgency",
        "addr": "ins1.clan-fa.com",
        "port": 27019,
        "channel":<PHONE_NUMBER>24297247,
        "country": "US",
        "frontMessage": "\n\n**Insurgency Games**\n Store : https://store.steampowered.com/app/222880/Insurgency/ \n ==================================================="
    },
    {
        "type": "SourceQuery",
        "game": "Insurgency",
        "addr": "ins2.clan-fa.com",
        "port": 27020,
        "channel":<PHONE_NUMBER>24297247,
        "country": "US"
    },
    {
        "type": "SourceQuery",
        "game": "Day of Infamy",
        "addr": "doi1.clan-fa.com",
        "port": 27030,
        "channel":<PHONE_NUMBER>24297247,
        "country": "US",
        "frontMessage": "\n\n**Day of Infamy**\n Store : https://store.steampowered.com/app/447820/Day_of_Infamy/ \n ==================================================="
    }
]

Contact
Discord : Spooker#0001<PHONE_NUMBER>17134080)

thanks!


Add token generation functionality

Functionality works, but might need some refactoring
• Assumes config file is JSON (template is in this PR)
• Only works with Gmail API at the moment
• Functions are ran from the same file they're defined (idk if this matters)

Color consts in debug statements are using + rather than , because commas cause a weird indentation
e.g.
console.debug(color + 'string') // no indent
console.debug(color, 'string') // indent

travis failed, build error

2nd Refactor:
• move setAuthCredentials() to AuthManager.js, it's only purpose is to get and write google api keys
• modularize GenerateTokenScript to outsource work to AuthManager to get API keys, and SceGoogleApiHandler to generate token


ISPN-11402 docs housekeeping

https://issues.redhat.com/browse/ISPN-11402
Needs backport to 10.1.x

@oraNod I found a missing link.
https://github.com/infinispan/infinispan/pull/8004/files#diff-d78699154ff27f3f49fbe1e11bf66961R17

no link for "limitations"

@oraNod I found a missing link.
https://github.com/infinispan/infinispan/pull/8004/files#diff-d78699154ff27f3f49fbe1e11bf66961R17
* no link for "limitations"

@pruivo thanks, should be fixed now. I think that was in the multimapcache docs.

integrated! thanks @oraNod !


Microsoft.Xaml.Interactivity bug in WinUI 3 Preview 4 making further development a non starter.

Describe the bug
Microsoft.Xaml.Interactivity is not working as expected with WinUI 3 Preview 4.
Given the following XAML:
            <Frame x:Name="ContentFrame" x:FieldModifier="Public" Padding="14 14" >
                <i:Interaction.Behaviors>
                    <ic:EventTriggerBehavior EventName="Loaded">
                        <ic:InvokeCommandAction Command="{x:Bind ViewModel.FrameLoadedCommand}" CommandParameter="{Binding ElementName=ContentFrame}"/>
                    </ic:EventTriggerBehavior>
                </i:Interaction.Behaviors>
                <Frame.ContentTransitions>
                    <TransitionCollection>
                        <NavigationThemeTransition>
                            <NavigationThemeTransition.DefaultNavigationTransitionInfo>
                                <DrillInNavigationTransitionInfo />
                            </NavigationThemeTransition.DefaultNavigationTransitionInfo>
                        </NavigationThemeTransition>
                    </TransitionCollection>
                </Frame.ContentTransitions>
            </Frame>

The CommandParameter is null when it should have the value of the Frame "ContentFrame" object.
Steps to reproduce the bug
You may use the following GitHub repository: https://github.com/Noemata/SimpleMVVM
Steps to reproduce the behavior:

Build SimpleMVVM
Set a breakpoint on the if () line as shown below within the SetupNavigationService method in ShellViewModel.cs which is part of the SimpleMVVM solution (note: UWP equivalent project is also provided).

        private void SetupNavigationService(Frame frame)
        {
            // MP! fixme: why is frame null??
            if (frame != null)
                NavigationService.Frame = frame;
        }

Click the About "( ! ) " menu item when the app launches.

Expected behavior
CommandParameter should not be null.
Screenshots

Version Info
NuGet package version: [Microsoft.WinUI 3.0.0-preview4.210210.4]
Windows app type:

UWP
Win32

No
Yes

Windows 10 version
Saw the problem?

October 2020 Update (19042)
Yes

Device form factor
Saw the problem?

Desktop
Yes

Additional context
This issue blocks any further development prospects with WinUI 3 Preview 4 as it cripples the ability to easily port working code over from UWP.  The provided sample has a fully functional UWP equivalent app using the exact same code constructs which should work in WinUI 3 Preview 4.

@Noemata There is a separate repo for Microsoft.Xaml.Interactivity here: https://github.com/microsoft/XamlBehaviors where you might have more luck reporting the problem you faced.

@duke7553 , I do not feel it is appropriate to start there, since this may turn out to be a WinUI 3p4 issue irrespective of the fact that it initially looks like a XamBehaviors problem.  I may post this there also.  Thanks for the suggestion.
I'm leaning toward blaming WinUI because this works in UWP.  So it suggests the framework XAML behaviors is sitting on may be broken.  Unless Xaml Behaviors is very different under the hood for WinUI (haven't looked myself).

@MikeHillberg FYI @JesseCol looks to have temporarily removed behaviors from XCG and might know more about the current state of its support.

Hi all, I removed XamlBehaviors from XCG a while back, I don't think it had been updated to work with WinUI 3 at that point.  But it looks like @azchohfi may be working on XamlBehaviors to keep up with the latest WinUI 3 previews -- maybe he knows if this is working well today?

@azchohfi, I tested a bunch of binding expressions with the Preview4 branch of XAML Behaviors.  Was any of this tested?  I see many things not working.  I can't tell what layer is at fault.
In future, please include release notes with XAML Behaviors so we know what to expect, if things are broken during your testing.  That said, this level of quality shouldn't get out the door.  Sorry to have to be so blunt.

This is an issue with WinUI3, when using Binding with ElementName. Not a Xaml Behaviors issue.
I believe @alwu-msft fixed this already, but its not out. The next public version should have this fixed. Alan, can you confirm?
This might be a dupe of another issue, but I couldn't find anything.

Correct, an upcoming preview of WinUI 3 should have the fix for the bug affecting ElementName Bindings. It seems that the recent discussion is leaning toward the theory that this is a dupe of the aforementioned bug, so I'm keeping my fingers crossed.

What's the status of this issue? Is any option to replace Microsoft.Xaml.Interactivity or we need to wait to another preview? @alwu-msft

@JanRajnoha , appears to be resolve with WinUI 0.5.  You can look for example usage here: https://github.com/Noemata/FakePOS

@Noemata It looks it is working. Thanks for tip

Closing this issue, since it is working in the Project Reunion 0.5 release as mentioned above.


Fix preprocess_data.py link

Fixing path for tools/datasets/preprocess_data.py

This PR removes the Lion argument from the list of supported optimizers. Was that deliberate?

This PR removes the Lion argument from the list of supported optimizers. Was that deliberate?

No, that was from the automatic: https://github.com/EleutherAI/gpt-neox/pull/1064/commits/68dceb081d462fbcf6833915d187ab4ac8a3b4d9
Haven't had a chance to fix it yet.


Error: Cannot find type 'UnityAPI' in scope

Hello! I used your blog post for Unity-SwiftUI integration, but I'm encountering an error in "UnityBridge.swift":

Appreciate any help!

That solved the issue. Thanks!


:cloud: upload to cloud rather than insert to mongodb

final commit before repo rename

:rocket: deployment of profiles-etl-combiner succeeded (http://profiles-etl-combiner-pr-7.dev.beta.nhschoices.net)

Changes Unknown when pulling cdca00e910dbe8b39d34c78ba3f69731c14940de on feature/merge-and-upload into ** on master**.

:rocket: deployment of profiles-etl-combiner succeeded (http://profiles-etl-combiner-pr-7.dev.beta.nhschoices.net)

:rocket: deployment of profiles-etl-combiner succeeded (http://profiles-etl-combiner-pr-7.dev.beta.nhschoices.net)

Changes Unknown when pulling 950563053f434db8590cd1613f815fca0b166b8c on feature/merge-and-upload into ** on master**.

:rocket: deployment of profiles-etl-combiner succeeded (http://profiles-etl-combiner-pr-7.dev.beta.nhschoices.net)


Ability.tickEvent clarification

Clarification that an ability's tickEvent does not behave the same way a normal Tick() function does, as it is considered an event.

Reworded it a bit but added, thank you!


Dolt processes large SQL statement as single line throwing error

I have a SQL file of the following form, it's about 11MB:
INSERT_INTO national_voting_data (`candidate`,`county`,`date`,`district`,`election`,`office`,`party`,`precinct`,`special`,`state`,`votes`,`year`) VALUES ('David King','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','REP',1,False,'CA',14,2014),
('David King','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','REP',1,False,'CA',14,2014),
('John Haggerty','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','REP',1,False,'CA',24,2014),
('John Haggerty','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','REP',1,False,'CA',24,2014),
('Jonathan Jaech','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','LIB',1,False,'CA',11,2014),
('Jonathan Jaech','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','LIB',1,False,'CA',11,2014),
('Kamala D. Harris','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','DEM',1,False,'CA',121,2014),
.
.
.

Running it through dolt sql results in the following error:
% dolt sql < test_file.sql
Rows inserted: 0 Rows updated: 0 Rows deleted: 0
bufio.Scanner: token too long

I can't upload the file to GitHub, as it's too large, but I can send it to you if need be, currently @andrew-wm-arthur has a copy.
This arose from the same query producing a bogus primary key error when run through the MySQL Python connector as a SQL statement.

There's a setting in Go you can change for this. I had to do it for a
single value but never a row.
On Mon, Jun 29, 2020 at 8:35 AM Oscar Batori<EMAIL_ADDRESS>wrote:

I have a SQL file of the following form, it's about 11MB:
INSERT_INTO national_voting_data (candidate,county,date,district,election,office,party,precinct,special,state,votes,year) VALUES ('David King','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','REP',1,False,'CA',14,2014),
('David King','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','REP',1,False,'CA',14,2014),
('John Haggerty','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','REP',1,False,'CA',24,2014),
('John Haggerty','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','REP',1,False,'CA',24,2014),
('Jonathan Jaech','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','LIB',1,False,'CA',11,2014),
('Jonathan Jaech','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','LIB',1,False,'CA',11,2014),
('Kamala D. Harris','Sutter','2014-06-03 00:00:00','NA','primary','Attorney General','DEM',1,False,'CA',121,2014),
..
.
..
Running it through dolt sql results in the following error:
% dolt sql < test_file.sql
Rows inserted: 0 Rows updated: 0 Rows deleted: 0
bufio.Scanner: token too long
I can't upload the file to GitHub, as it's too large, but I can send it to
you if need be, currently @andrew-wm-arthur
https://github.com/andrew-wm-arthur has a copy.
This arose from the same query producing a bogus primary key error when
run through the MySQL Python connector as a SQL statement.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/liquidata-inc/dolt/issues/775, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABJAR3DYIISFSEJHCI6Z24TRZCYCPANCNFSM4OLKYIJQ
.

@timsehn is there a way that we can make that the default behavior?

Yeah. Change the Dolt code. Like we need to modify the Dolt code to accept
bigger statements.
On Mon, Jun 29, 2020 at 11:18 AM Oscar Batori<EMAIL_ADDRESS>wrote:

@timsehn https://github.com/timsehn is there a way that we can make
that the default behavior?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/liquidata-inc/dolt/issues/775#issuecomment-651281626,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABJAR3GGECH4GXPDOOP43RLRZDLILANCNFSM4OLKYIJQ
.


Understanding Sharing Keras Layers

I'm trying to build a variational autoencoder that encodes and decodes music sequence data (of shape (50, 84)). Simultaneously, I want to predict the next element inside the music sequence (vector of shape (84,)). So I need to reuse a trainable decoder that maps from the latent space to the output space – for both encoding and next element prediction.
autoencode_output = TimeDistributed(decoder) (z)

x = GRU(64, dropout_W=0.1, dropout_U=0.1, return_sequences=True) (z)
x = GRU(12, dropout_W=0.1, dropout_U=0.1, return_sequences=False) (x)
next_output = decoder(x)

model = Model(input_img, [autoencode_output, next_output])

This is the relevant part of the code; I reuse the "decoder" model twice (in the TimeDistributed wrapper and also by itself). I keep on getting a 'this shared variable already has an update expression' error, however. When do these errors happen when reusing models, and what's the cleanest way to fix them?
Thanks!

question more suitable for stack overflow

Hello, I tried to reproduce your bug but couldn't :
def buildDecoderModel():
inp = Input(batch_shape=(None,100))
decout = Dense(10)(inp)
decoder = Model([inp],[decout] )
inp2 = Input(batch_shape=(None,30,100))
emb = TimeDistributed( decoder )(inp2)
gru = GRU(100,return_sequences=True)(emb)
gru2 = GRU(100,return_sequences=False)(gru)

dec = decoder(gru2)

model = Model( [inp2],[gru,dec])
model.compile( loss=["mse","mse"], optimizer="adam" )
return model

I tested with theano :
import numpy as np
m= buildDecoderModel()
m.fit(np.random.randn(10,30,100),[np.zeros((10,30,100)) , np.zeros((10,10)) ] )
It works fine.


add test case about cluster clusterrole

Signed-off-by: ldpliu<EMAIL_ADDRESS>https://github.com/stolostron/backlog/issues/25657

/lgtm

/retest

/retest

/retest

/retest


Specifying logical signal direction for ports of an elaboratable

 Issue by ZirconiumX
Monday Sep 30, 2019 at 12:53 GMT
Originally opened as https://github.com/m-labs/nmigen/issues/243

This came up on IRC a while back, and I thought I'd write it down for the sake of posterity.
Signals should have an optional must_read and must_write parameter for linting purposes. This would be used to mark Signals intended for external use by other Elaboratables, and warn (error?) when they are not used correctly; for example, a FIFO's input would be marked must_write and its output would be marked must_read, because if the input was not written to, or the output not read from, the FIFO would be useless.
I would suggest semantics for these as must_read and must_write producing a warning if they are never used as the source, or destination of a connection between Signals (I don't know the nMigen term for this).
Perhaps one could go further and warn if an Elaboratable does not declare any of its Signals must_read or must_write, as their linting purpose is unhelpful if all signals are optional. This would likely be quite noisy on an established nMigen codebase however.

This was done as a part of RFC 2.


Merged Cookies On Self Hosted

Set-Cookie headers

Hey,
Could you please elaborate a bit on this? Isn't Cookies.Add working like it should? We have a faint memory of seeing something about it but could not find any info about it.

Andreas, I apologize for not explaining more in my github pull request.  I
posted on the Nancy Google Group earlier yesterday regarding this issue.
I encountered the issue when I added multiple cookies to a Nancy Self
Hosted project.  Fiddler reported a response with only one cookie that had
all of my cookies comma separated.  However within chrome and firefox the
merged cookie only had the CSRF token as well as the first cookie I added.
(I haven't verified this but I believe this has something to do with
HttpOnly cookies.) Getting frustrated I finally tracked this down to a
Microsoft .Net hotfix
(http://support.microsoft.com/kb/933905http://www.google.com/url?sa=D&q=http://support.microsoft.com/kb/933905&usg=AFQjCNFgxK3f17hDkUO-iMgio2roVwaQ9g).
At the bottom it said to use the HttpListnerResponse.Headers.Add method.
BTW, Andreas, you guys have really done a superb job with Nancy.  I love
it's simplicity and how well thought out the codebase is.  Keep up the good
work.  If you guys need anything let me know and I can definitely
contribute.
Cheers,
Cedric
2011/11/15 Andreas Hkansson <
<EMAIL_ADDRESS>

Hey,
Could you please elaborate a bit on this? Isn't Cookies.Add working like
it should? We have a faint memory of seeing something about it but could
not find any info about it.

Reply to this email directly or view it on GitHub:
https://github.com/NancyFx/Nancy/pull/407#issuecomment-2743003


ForeachElement Does not Re-prompt

Version
4.16.0
Describe the bug
We are running an InputDialog inside an Adaptive For each, another dialog is then run proactively when that proactive dialog has finished then input dialog resumes and Re-prompts the user where they were left before.
Since upgrading to 4.16.0 we noticed that ForEach is Obsolete using the new ForEachElement the above behaviour no longer exists and the InputDialog is not re-prompted.
Expected behaviour
The same re-prompting behaviour to be in the ForEachElement that was in the older ForEach.
Screenshots
Conversation with the old ForEach :

Here is 4.16.0 ForEachElement without re-prompt:

Hey @karrimrabibiomni -- Thanks for reaching out.
For clarity, could you please share the steps to reproduce this issue? I will attempt to reproduce from my end.
Also, are you able to share your dialog code?

Hi @karrimrabibiomni
Thank you for the repro project! This PR adds reprompt support to ForEachElement, with a unit test of the repro you've provided: https://github.com/microsoft/botbuilder-dotnet/pull/6325


Convert unicode bytes to utf-8 strings

close #581

Thanks! I tidied it up a bit to use unicode codepoint literals instead of byte strings w/ .decode('utf8'). Should be completely equivalent but LMK any concerns.
Merging...


Master

Add institution (publisher) functionality for GBIF

Add CSV write functionality for publishers data

New folders for CSV (sources and written)

Renamed files: main.py -> main_GBIF.py, test.py -> csv_functions.py

Tweaks to existing functions

For next time check if you can get the format of the branch to something like: "feature/{minimal description of functionality}"
For a bug fix we can then use bug/{}
I searched for the git branching name conventions which this comes from but was unable to find it.
However if we do this consistently we should be fine

Add a .gitignore file to your repo (top level).
This will ensure that files that you don't want to commit (like the .idea folder) are passed in the git commit.
https://git-scm.com/docs/gitignore (can also check a repo of mine)

I would probably remove the written folder as this should fill itself when to code has been run. It is not part of the code and can change each time you run it (so it isn't stable). Maybe have the directory there but have it empty.

Add a README.md file (which is in Markdown language https://www.markdownguide.org/) To describe to a user how they can run the application

Add a requirements file (https://realpython.com/lessons/using-requirement-files/) to the project so the version of your libraries are fixed.

Check if you can get a clear main() method which start the script and runs through all the methods. I now find it a bit hard to see how it is all connected. I thought all logic was in the main_GBIF file and the csv_functions only write to the csv.
Then the flow would be:
main_GBIF -> main function -> first data retrieval -> data logic -> data write -> second data retrieval -> logic -> write, etc...
You could even split up these four parts, one file with the main, one file with the retrieval, one with the logic, one with the write.
Splitting the logic from the retrieval would also help with writing unit tests later.
Note that for not this splitting up is not really necessary, make sure there is a main() which calls all the functions and makes it into a nice executable program and only when there is time left do the refactoring

For next time check if you can get the format of the branch to something like: "feature/{minimal description of functionality}" For a bug fix we can then use bug/{} I searched for the git branching name conventions which this comes from but was unable to find it. However if we do this consistently we should be fine

I think you mean something like the following article states: https://gist.github.com/digitaljhelms/4287848? So when I would develop a new functionality, I will need to create a local branch of master that is named after the feature or id in TeamWork. This way it is easier to see how changes were implemented. They also call out Bug and HotFix branches.

Add a .gitignore file to your repo (top level). This will ensure that files that you don't want to commit (like the .idea folder) are passed in the git commit. https://git-scm.com/docs/gitignore (can also check a repo of mine)

I have added a .gitignore file to the following push for .idea and pycache, not sure if this will be part of this pull request or a new one (do not exactly know how this flow functions).

I would probably remove the written folder as this should fill itself when to code has been run. It is not part of the code and can change each time you run it (so it isn't stable). Maybe have the directory there but have it empty.

Do you mean the contents of the folder? At the moment the csv files are being overwritten each time. I recalled hearing that we could store the csv by year and month, would that be an executable idea?

Add a README.md file (which is in Markdown language https://www.markdownguide.org/) To describe to a user how they can run the application

Will be included in the next commit / push

Add a requirements file (https://realpython.com/lessons/using-requirement-files/) to the project so the version of your libraries are fixed.

Added requirements file for the next commit/push

Check if you can get a clear main() method which start the script and runs through all the methods. I now find it a bit hard to see how it is all connected. I thought all logic was in the main_GBIF file and the csv_functions only write to the csv. Then the flow would be: main_GBIF -> main function -> first data retrieval -> data logic -> data write -> second data retrieval -> logic -> write, etc... You could even split up these four parts, one file with the main, one file with the retrieval, one with the logic, one with the write. Splitting the logic from the retrieval would also help with writing unit tests later.
Note that for not this splitting up is not really necessary, make sure there is a main() which calls all the functions and makes it into a nice executable program and only when there is time left do the refactoring

I have simplified the main file so every function can be called from a core function called 'main'. Code and file logic has been adjusted to this approach. A future installment could also separate the data retrieval and logic.

Starting to look good! I think we can tighten a couple of spot but let's do that in the next round of review. Let's let this pass so we can move on with the story and the task can move to the completed lane before the sprint planning


Very slow with type signatures

The tests take about 9 seconds in the attached file take about 9 seconds to run.
If I remove the type signature for inorder, execution time drops to 1,1s
If I additionally remove the type signature for the constructor, times drops to 0,3s

from __future__ import annotations
import unittest
from wypp import *

@dataclass
class SearchTreeNode:
    key: any
    value: any
    left: Optional[SearchTreeNode]
    right: Optional[SearchTreeNode]

class SearchTree:
    def __init__(self, root: SearchTreeNode = None):
    #def __init__(self, root = None):
        self.root = root

    # SEITENEFFEKT: verändert den Suchbaum
    def insert(self, key: any, value: any):
        self.root = self.__insertAux(self.root, key, value)

    def __insertAux(self, node: Optional[SearchTreeNode], key: any, value: any) -> SearchTreeNode:
        if node is None:
            return SearchTreeNode(key, value, None, None)
        elif node.key == key:
            node.value = value
            return node
        elif key < node.key:
            node.left = self.__insertAux(node.left, key, value)
            return node
        else: # key > node.key
            node.right = self.__insertAux(node.right, key, value)
            return node

    def inorder(self) -> list[tuple[any, any]]:
    #def inorder(self):
        result = []
        self.__inorderAux(self.root, result)
        return result

    def __inorderAux(self, node: Optional[SearchTreeNode], result: list[tuple[any, any]]):
    #def __inorderAux(self, node, result):
        if node is not None:
            self.__inorderAux(node.left, result)
            result.append( (node.key, node.value) )
            self.__inorderAux(node.right, result)

# Tests
import random

def randomInts(n, low, high):
    res = []
    for _ in range(n):
        i = random.randint(low, high)
        res.append(i)
    return res

class SearchTreeTest(unittest.TestCase):

    def assertSearchTree(self, node):
        if node == None:
            return
        else:
            for k,_ in SearchTree(node.left).inorder():
                self.assertTrue(node.key > k)
            for k,_ in SearchTree(node.right).inorder():
                self.assertTrue(node.key < k)
            self.assertSearchTree(node.left)
            self.assertSearchTree(node.right)

    def measure(self, start, times, i):
        import time
        end = time.time()
        times[i] = times[i] + (end - start)
        return end

    def testIsSearchTree(self):
        times = [0,0,0]
        import time
        for _ in range(100):
            start = time.time()
            l = randomInts(30, 0, 50)
            t = SearchTree()
            for i in l:
                start = self.measure(start, times, 0)
                t.insert(i, "value_" + str(i))
                start = self.measure(start, times, 1)
                self.assertSearchTree(t.root)
                start = self.measure(start, times, 2)
        print(times)

if __name__ == '__wypp__':
    unittest.main(argv=['ignored'], exit=False, verbosity=2)
elif __name__ == '__main__':
    unittest.main()

The problem is in inspect.Signature.bind
In was able to implement a version that is about 20%-25% faster. Still unsure if this improvement is enough to justify duplicating quite complex logic from the inspect module.

See #104 for a pragmatic solution.
Here are the performance numbers:
# Measured 2022-05-05 on a Macbook pro with a 2,3 GHz 8-Core Intel Core i9
#
# * Without type annotations
#
# Run 1
# ['0.004', '0.258', '0.054']
# Run 2
# ['0.004', '0.258', '0.054']
# Run 3
# ['0.005', '0.262', '0.055']
# Median of total: 0.317
#
# * With type annotations (slow way)
#
# Run 1
# ['0.005', '0.271', '2.548']
# Run 2
# ['0.005', '0.272', '2.509']
# Run 3
# ['0.005', '0.271', '2.526']
# Median of total: 2.803
# Factor: 8.8
#
# * With type annotations (fast way)
# Run 1
# ['0.005', '0.126', '1.128']
# Run 2
# ['0.005', '0.120', '1.072']
# Run 3
# ['0.005', '0.119', '1.073']
# Median of total: 1.197
# Factor: 3.7

Upshot: wrapping arguments is still quite expensive. But in the common case (no kw arguments, no variable arguments, no pre-/postcondition), the factor is no longer 8.8 but only 3.7.


Unicode literal range

The documentation states that:

... or UTF-32 encodings that you can specify by using \U followed by a 32-bit hexadecimal code that represents any Unicode code point (00000000 - 00010FFFF).

However the upper bound seems wrong here, specifically the literal "\U00010FFFF" results in a 3 character string, with the third character being 'F'.
Unless I'm mistaken,  the upper bound should be 0010FFFF.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e3a3edd2-5887-40c7-4feb-0b0cd71f4b8a
Version Independent ID: 031fc9cd-677b-25a5-f708-f2c2f991e61a
Content: Literals - F#
Content Source: docs/fsharp/language-reference/literals.md
Product: dotnet-fsharp
GitHub Login: @cartermp
Microsoft Alias: phcart

This is fixed in #13257 - I don't think the merge to live went through yet.