Ingress capabilities detection

Hi,
Since we upgraded our Kubernetes cluster we identified that the kong.ingressVersion helper function is not applying the right capabilities detection switch.
We are automatically detecting capabilities on our cluster and passing those to the Helm templating via kubectl api-versions
Current implementation defines
{{- define "kong.ingressVersion" -}}
{{- if (.Capabilities.APIVersions.Has "networking.k8s.io/v1/Ingress") -}}
networking.k8s.io/v1
{{- else if (.Capabilities.APIVersions.Has "networking.k8s.io/v1beta1/Ingress") -}}
networking.k8s.io/v1beta1
{{- else -}}
extensions/v1beta1
{{- end -}}
{{- end -}}

Where it should be
{{- define "kong.ingressVersion" -}}
{{- if (.Capabilities.APIVersions.Has "networking.k8s.io/v1") -}}
networking.k8s.io/v1
{{- else if (.Capabilities.APIVersions.Has "networking.k8s.io/v1beta1") -}}
networking.k8s.io/v1beta1
{{- else -}}
extensions/v1beta1
{{- end -}}
{{- end -}}

Are our assumptions correct ?
If so I will raise a PR for this.
Many thanks.

Hi @ludovic-pourrat 👋
It seems you're right. Given this handy template:
apiVersion: v1
kind: ConfigMap
metadata:
  name: apiversions-{{ .Release.Name }}
  namespace: apiversions
data:
  apiVersions.txt: |
    {{- range .Capabilities.APIVersions }}
    - {{ . }}
    {{- end -}}

I can confirm I'm getting the api versions as you've indicated:
---
# Source: kong/templates/cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: apiversions-kong-test1
  namespace: apiversions
data:
  apiVersions.txt: |
    - v1
    - admissionregistration.k8s.io/v1
    - admissionregistration.k8s.io/v1beta1
    - internal.apiserver.k8s.io/v1alpha1
    - apps/v1
    - apps/v1beta1
    - apps/v1beta2
    - authentication.k8s.io/v1
    - authentication.k8s.io/v1beta1
    - authorization.k8s.io/v1
    - authorization.k8s.io/v1beta1
    - autoscaling/v1
    - autoscaling/v2
    - autoscaling/v2beta1
    - autoscaling/v2beta2
    - batch/v1
    - batch/v1beta1
    - certificates.k8s.io/v1
    - certificates.k8s.io/v1beta1
    - coordination.k8s.io/v1beta1
    - coordination.k8s.io/v1
    - discovery.k8s.io/v1
    - discovery.k8s.io/v1beta1
    - events.k8s.io/v1
    - events.k8s.io/v1beta1
    - extensions/v1beta1
    - flowcontrol.apiserver.k8s.io/v1alpha1
    - flowcontrol.apiserver.k8s.io/v1beta1
    - flowcontrol.apiserver.k8s.io/v1beta2
    - networking.k8s.io/v1
    - networking.k8s.io/v1beta1
    - node.k8s.io/v1
    - node.k8s.io/v1alpha1
    - node.k8s.io/v1beta1
    - policy/v1
    - policy/v1beta1
    - rbac.authorization.k8s.io/v1
    - rbac.authorization.k8s.io/v1beta1
    - rbac.authorization.k8s.io/v1alpha1
    - scheduling.k8s.io/v1alpha1
    - scheduling.k8s.io/v1beta1
    - scheduling.k8s.io/v1
    - storage.k8s.io/v1beta1
    - storage.k8s.io/v1
    - storage.k8s.io/v1alpha1
    - apiextensions.k8s.io/v1beta1
    - apiextensions.k8s.io/v1

Feel free to submit a fix for that.

@ludovic-pourrat thanks for reporting :bow:.
Would you be interested in submitting a PR?

@ludovic-pourrat
After discussion with the team: before jumping to a fix, could you please let us know what exact issue has this caused for you?

BTW it seems that we can't query particular objects APIs via .Capabilities.APIVersions
e.g. having:
$ k get --raw /apis/networking.k8s.io/v1 | jq '.resources[] | select(.kind == "Ingress")'
{
  "name": "ingresses",
  "singularName": "",
  "namespaced": true,
  "kind": "Ingress",
  "verbs": [
    "create",
    "delete",
    "deletecollection",
    "get",
    "list",
    "patch",
    "update",
    "watch"
  ],
  "shortNames": [
    "ing"
  ],
  "storageVersionHash": "39NQlfNR+bo="
}
{
  "name": "ingresses/status",
  "singularName": "",
  "namespaced": true,
  "kind": "Ingress",
  "verbs": [
    "get",
    "patch",
    "update"
  ]
}

{{ (.Capabilities.APIVersions.Has "networking.k8s.io/v1/Ingress" ) }} returns false

@ludovic-pourrat
After discussion with the team: before jumping to a fix, could you please let us know what exact issue has this caused for you?

We cannot deploy anymore Kong because the default to the release of betav1 is not anymore available in our migrates cluster.

Am aside note we are running Kubernetes 1.23 where beta/v1 is dropped.

Not sure I follow your question about the built-in versus custom API versions. Are you referring to the Helm behavior or to the fakegroup.example.com/v1 version?
I included the fakegroup.example.com/v1 to demonstrate the difference between Helm's behavior for standard Kubernetes groups (Helm appears to have built-in info about assumed available groups when running helm template without any --api-versions arguments) and custom ones. The custom ones aren't actually relevant here, though we do use them elsewhere, e.g. to detect whether we can create Prometheus resources.
We've kept support for 1.19 resources in part out of a lack of compelling reason to remove them and in part because we do observe a number of still active 1.19 deployments in the wild. Absent something else that will break compatibility, we tend to try and keep compatibility with with older versions if possible, and it was easy to do so here because we'd already written the template logic when 1.19 was still a supported version.
The Helm built-in knowledge is also incomplete: it's aware of the group and version (though it's not clear what Kubernetes version it bases these on), but not individual kinds within it, and Kubernetes can introduce new beta version kinds into an existing group that has v1 APIs already. The built-in versions furthermore aren't guaranteed to match the actual destination cluster's, since they're determined entirely by the Helm version. IMO the ideal solution here is for the major users of template (various CD solutions) to find a way to essentially mimic the data normally available to Helm during install runs and populate this automatically--individual chart authors shouldn't need to design for two modes of operation that function differently. Helm has more or less indicated that they'd prefer template users switch to an SDK-based implementation rather than adding additional template CLI functionality.

@rainest,
Sorry I was not clear enough in my explanations for the built-in options in Helm, and I was referring the one when you pass the api-versions to the templating.
Our deployment process execute a kubectl api-versions and pass it to Helm in order to template the files as we are using Argo  for the deployment.
What is not clear to me is how this is supposed to work in a Kubernetes release greater than 1.19 ? (if not using the api-versions).

This sort of issue only appears with helm template, not with helm install. The latter is what we've historically targeted, because that is the standard means of installing a Helm chart. My understanding of the situation (to be taken with a grain of salt--I haven't discussed this with the stakeholders) is roughly:

Helm was initially designed with install as the install tool and template as mostly a debugging utility.
Various CD tools (Argo ageneralnd such) started adding Helm install methods because of the wide availability of charts for applications. Because they wanted to manage applying resources themselves, they used helm template instead of helm install.
template isn't really well-suited for this purpose, because it does a lot of things you probably wouldn't want for it, namely rendering test and lifecycle hook templates by default, not applying the namespace, and breaking anything that requires talking to the target Kubernetes cluster.
The Helm project does not want to add all functionality needed to make template work well for the CD case to the command, because they consider it a tool designed for humans, whereas integrations should instead use the Helm Go API (which should essentially allow them a path to doing the useful things install does without losing control over resource installation).

The two parties have not resolved this, so we (chart authors) end up stuck in the awkward situation of needing to either offer annoying workarounds or add additional toggles to our values.yaml to replicate functionality normally provided by Helm.


Commented out git branch track main

Remove git branch --track main origin/main as it failed to run in main

@kai-nguyen-aligent what does this line do? There's a history of it being removed and added back again :sweat_smile:

To be honest, I can't remember what it does anymore. I was fix this and that around this part several times. On possible reason is it was added before adding the 2 lines that actually fix the issue:
`- uses: nrwl/nx-set-shas@v4

run: git fetch`
Now that these 2 line are added, this one is not need anymore.


db connection

thank you for writing catapult and raven ! I was just wondering if there is an easy way to connect to a database like mysql ? If you need help in terms of sponsoring features let me know. I might be able to donate to the project.

the binding of mysql is on working!
i will publish in raven since i have a stable version

Catapult is not the principal
but Igropyr is:
https://github.com/guenchi/Igropyr

@gurugeek mysql binding is ok
raven install mysql
or
https://github.com/chclock/mysql


Protection path for ConnectivityService

Hello,
Unfortunately from the TAPI description I did not find the possibility to create a protection path for ConnectivityService.
Could TAPI community throw a light on the mentioned issue?

Protection use cases are in progress in ONF. And then will deal with protection issue of ConnectivityService.

@RodLU is right. Explicit control of protection/restoration is under discussion and scheduled for next major release.

@RodLU, @karthik-sethuraman thank you for explanation. Could you please share a possible date of a new TAPI major release?

The TAPI Resilience feature draft is now part 2.0


OSError: [Errno 10048]

Problem Description
When i am trying to open mitmproxy it is showing this error
OSError: [Errno 10048] error while attempting to bind on address ('<IP_ADDRESS>', 8080): only one usage of each socket address (protocol/network address/port) is normally permitted
here is log
Traceback (most recent call last):
File "mitmproxy\master.py", line 54, in run_loop
File "urwid\main_loop.py", line 287, in run
File "urwid\main_loop.py", line 385, in _run
File "urwid\main_loop.py", line 1494, in run
File "urwid\compat.py", line 58, in reraise
File "mitmproxy\addons\proxyserver.py", line 154, in refresh_server
File "asyncio\streams.py", line 94, in start_server
File "asyncio\base_events.py", line 1494, in create_server
OSError: [Errno 10048] error while attempting to bind on address ('<IP_ADDRESS>', 8080): only one usage of each socket address (protocol/network address/port) is normally permitted
mitmproxy has crashed!
System Information
Mitmproxy: 7.0.4 binary

There's probably already something listening on port 8080 on that machine (a different service or a different instance of mitmproxy), try a different port via --listen-port parameter. This doesn't look like a bug in mitmproxy, so I'm closing this.


Remove Config.required_fields

This parameter on Config has its origins in work done to parse
"includes" shown here: c794148465722369a562af7f5d521b2d51d20b07.
As far as I can tell this parameter is both undocumented and unused. It
should be completely safe to remove.

@jeregrine assigning you to review this one. You're the author of the original commit. Perhaps you can shine some light on this mystery param?

@jeregrine have you had any time to mull this over?

Sorry for delay. Looks good.


Research a library to select colors for slices if possible within our existing code

References #32

Started research.
[t:90]

More looking around.
[t:90]

It seems there is no way to change just the area that is sliced, either for opacity or color, with how this is currently implemented due to Ipyvolume only allowing a single volume at a time.
It is possible that a different visualization type could be used, such as meshes, with the sliced halves being different meshes, which can be graphed differently. However, I don't believe there's a way to modify just the impression left of the slice plane.
As far as 3D slicing goes, I don't think there is any option that could reasonable replace ipyvolume, especially this late in the game. I think our best bet is to focus on 2D slices as requested by the sponsor.
Research done.
[t:30]


Try using debian:8 for older glibc

Building on debian:10 results in a binary that requires glibc 2.27 which is not available on some LTS server Linuxes like CentOS 7. This commit attempts to make a compatible jffi.so using debian:8, which ships with glibc 2.19. This is not as old as 2.17 but may remove enough dependencies on newer glibc to be sufficient.
See #138

This did not appear to pull in an older glibc, so back to the drawing board unfortunately. I tested on CentOS 7 and it still tries to find the GLIBC_2.27 symbol, so the debian:8 image must be on a newer glibc than it was originally released with.


Pass tests with the mysql2 adapter

Current build will trigger multiple failures, including:
<#<NoMethodError: undefined method `copy_table' for #<ActiveRecord::ConnectionAdapters::Mysql2Adapter:0x00000104138ff8>>>.

and
RuntimeError: Unsupported adapter ActiveRecord::ConnectionAdapters::Mysql2Adapter for this destination

Fixed: adapter_extensions rewritten, tests fixed. Closing.


Have options autocomplete for shell tool

Would be nice to have options autocomplete for the shell flutter tool in bash, zsh, fish.
Something like:

I feel like someone looked into this at some point.  There may even have been some code for this?  Maybe @devoncarew or @jimbeveridge remember?

@danrubel discussed this (he added similar support for fletch I believe).

Very brief research. But it seems like we can either:

Program it in via the the built-in complete and insert the various options one by one. We can potentially do it when building the flutter tool.
Somewhat a slight superset of 1 but let complete refer to a more complex shell function that's contextually smarter.
Contribute to some widely maintained 'framework' for completions like https://github.com/zsh-users/zsh-completions/tree/master/src and https://github.com/fish-shell/fish-shell/tree/master/share/completions.
Create man pages for flutter(potentially autogen'ed from --help). zsh, fish can scrape man pages for completions.

In ascending order, the cost is probably 1<4<3<2

I like auto-generating a tab complete shell script when building the flutter tool so that it stays in sync with flutter tool itself. Everything needed should already be in the flutter tool itself including text for auto generating the man pages.
When implementing cmdline tab completion for fletch / dartio, I wanted to minimize the shell script so I tried having a shell script launch a dart app to provide tab completion, but the result felt sluggish. I think code gen when building the flutter tool is the way to go.
Working on this interests me, but I have too many other things on my plate right now.

Seems like 'installing' man pages (4) reliably for all OSes isn't easy without root and we won't be running flutter script in root. Doesn't seem like a good vehicle unless the user installs with homebrew (and homebrew is installed on /usr/local with chmod to the user) or apt-get.
(1) is rather incomplete. Having sub-command's options and their own sub-commands' autocomplete is pretty useful too.
I think it's back to (2) and (3). Not an easy fix.

This might make this easier - https://github.com/kevmoo/completion.dart/
Assuming you're using 'pkg/args'

Waaaaat. Was just thinking whether the Dart sdk could include something like https://github.com/alecthomas/kingpin#bashzsh-shell-completion. Will take a look.

@xster haven't touched this in a VERY long time. Would love another set of eyes on it.

Duping #18988 which has more recent activity

Duping #18988 which has more recent activity

This issue mentions fish while #18988 does not. Both issues are now closed. Does it mean fish completions will not be supported?

@shilangyu can you open an issue specifically for fish?

@xster u can try my solution: https://github.com/qiuxiang/flutter-fish-completions write with dart, parse and generate fish completions from flutter help command.


[$500] Android - IOU-In a address, one word in Hindi and other text in English are shown.

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.3.90-1
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:

Launch app
Tap fab---request money--Distance
Tap start
Enter Chandigarh railway
Select suggested address "Chandigarh Railway Station Ticket Counter, Gali Number 2, Daria, Chandigarh, India"
Tap Start

Expected Result:
The address in address field and suggested addresses with text in Hindi language must not be shown.The entire address must only be in English language. In a address, one word in Hindi and other text in English must not be shown
Actual Result:
The address in address field and suggested addresses showing address with text in Hindi language. In a address, one word in Hindi and other text in English are shown
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos

Android: Native

https://github.com/Expensify/App/assets/78819774/b4cb5f04-7d37-42f3-947f-e0a7dcc0c6e0

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~015ca2087437191a6c
Upwork Job ID:<PHONE_NUMBER>696782336
Last Price Increase: 2023-10-24

Waiting for proposals

Will be changed here: https://github.com/Expensify/App/pull/29931

@0xmiroslav thoughts on the proposal above?

@0xmiroslav I just added it as a a heads up that it may 'change' the behaviour here I didn't say it would get fixed...

Still waiting on proposals

I think we're still waiting for proposal feedback, is that right?

Thanks!

Makes sense to me 👍

PR still in review

Thanks for that @0xmiroslav. Adding a payment summary and issuing payments via Upwork:

Issuer reporter: QA
Contributor: @tienifr $500
Contributor+: @0xmiroslav $500

All contracts paid out. Closing!


Describe error The remote virtual network lacks a gateway

Through a support case we actually got confirmed this theory - we were told it's Portal bug but I see it rather as a Portal limitation. Maybe we could get more precision on this one, but I find it important to have the error message listed anyway, as the solution indeed applies.
I was also thinking to link to Docs about how to enable (mention the -useRemoteGateways and --use-remote-gateways parameters), but I only see the list of parameters in Manage vnet peering but no PowerShell/CLI details, and I'm not sure if this level of detail is desired?

@Tbohunek : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#sign-off

@v-miegge no suggestions on documenting the required switches? :(

@Tbohunek: I do not have that technical knowledge, as I am only listed as the "author", since I entered the document into GitHub. But Kaushik Ainapure<EMAIL_ADDRESS>sent me the information to create the document. I would contact him regarding the technical aspect of the information in question.

@ktoliver can we update the wording a little bit more? From your edited version it's not clear that the feature must be enabled during creation. It cannot be enabled after the peering is created, and I think this needs to be explicitly highlighted.
Thanks.

@v-miegge - please update the wording to clarify that feature should be enabled during creation.


Attaching Control under Smoothing2D causes assert error: _SetProcessing(): Condition "!viewport" is true on set_as_top_level

I know it's not common to attach a Control of a PhysicsBody2D, but I have a local HUD displaying life gauge near attached to my CharacterBody2D so it's convenient, and I need to smooth it or it will jitter relatively to my smoother character sprite.
(usually I'd use a Canvas Layer and manually move the gauge to match character, but it's more complex)
When you do so, you get a double assert error on start: _SetProcessing(): Condition "!viewport" is true on set_as_top_level:
Godot code:
func _SetProcessing():
	var bEnable = _TestFlags(SF_ENABLED)
	if _TestFlags(SF_INVISIBLE):
		bEnable = false

	set_process(bEnable)
	set_physics_process(bEnable)

	set_as_top_level(_TestFlags(SF_GLOBAL_OUT))  # here

Callstack:

E 0:00:01:0024   smoothing_2d.gd:110 @ _SetProcessing(): Condition "!viewport" is true.
<C++ Source>   scene/gui/control.cpp:2984 @ _notification()
  smoothing_2d.gd:110 @ _SetProcessing()
smoothing_2d.gd:57 @ @flags_setter()
smoothing_2d.gd:234 @ _ClearFlags()
smoothing_2d.gd:245 @ _ChangeFlags()
smoothing_2d.gd:121 @ _notification()

(this shows twice)
However, the Control is still smoothed correctly, although set_as_top_level failed, apparently because it's called again later.

Could be fixed by #39, #40 , but will wait to here from OP before closing.

Indeed, error didn't appear as I was detaching Smoothing Node and reattaching it to an external parent node already in v1.0.3, but I also tested v1.2.1 disabling legacy toplevel and using global out with no Smoothing Node detachment at all, and it also works without error.


Updated compose cli to check for both http and https protocol

This change includes fix for compose cli to handle ssl enabled stacks and also updated readme for swarm to override PROTO environment variable.

Updated the README file and squashed the changes in one commit.

LGTM.


Update README.md [ci skip]

This PR:

remove outdated anchor link
link to rouge docs for CSS

view updated readme

Ah! Thanks!


How to send duplicate keys in post data?

I need to send a slightly unusual request with some repeating keys in the post data - like foo=1&foo=2. The target server requires it like this and normal foo[]=1&foo[]=2 won't work.
While googling for a solution using guzzle I found this documentation for old guzzle. Clearly such request was easy to make in the old version using the Guzzle\Http\QueryAggregator\DuplicateAggregator.
How can I do this in the new guzzle? My search was unsuccessful.

I guess you can just post like this array data.
foo[0]=1
foo[2]=2
foo[3]=1

No I can't. And I already said so in the first post.

Guzzle internally just passes form_params array to PHP's http_build_query() function. So no way.
IMO the only option now is to do it manually, something like that:
$params = [
    ['foo' => 'bar'],
    ['foo' => 'baz'],
];

$body = [];
foreach ($params as $param) {
    $body[] = http_build_query($param);
}

$body = implode('&', $body); // string(15) "foo=bar&foo=baz"

@alekseykuleshov So this feature was removed without replacement when upgrading guzzle? Good to know. RIP Guzzle then.

FYI - This thread shows an alternative function which can be used called build_query.  Someone also showed example usage in the same thread.

$response = $client->request('POST', $url, [
    'multipart' => [
        [
            'name'     => 'text',
            'contents' => 'Hello'
        ],
        [
            'name'     => 'text',
            'contents' => 'Hi'
        ],
        [
            'name'     => 'text',
            'contents' => 'How are you'
        ]
    ]
]);

Use name and contents when you want to send duplicate keys in form data


Some Cassini ISS images marked for Saturn are really of Saturn Rings

This search (from the API result count test):
https://tools.pds-rings.seti.org/#/instrument=Cassini+ISS&surfacegeometrytargetname=SATURN&SURFACEGEOsaturnsubobserverplanetographiclatitude1=50&SURFACEGEOsaturncenterresolution1=5&SURFACEGEOsaturncenterresolution2=10&target=SATURN&view=detail&browse=gallery&colls_browse=gallery&page=1&gallery_data_viewer=true&limit=100&order=time1&cols=opusid,instrument,planet,target,time1,observationduration&widgets=instrument,surfacegeometrytargetname,SURFACEGEOsaturnsubobserverplanetographiclatitude1,SURFACEGEOsaturncenterresolution1,target&widgets2=&detail=co-iss-n1545019067
shows that the TARGET is Saturn, but the image is clearly of Saturn Rings. If you look at the label file:
https://pds-rings.seti.org/holdings/volumes/COISS_2xxx/COISS_2027/data/1545009755_1545083950/N1545019067_1.LBL
you'll see that TARGET_DESC is "Saturn-Rings".
Perhaps in some cases other than "Sky" we should let TARGET_DESC override TARGET?

Oh, the sordid tale!
I was part of an extended conversation on this topic that involved people from the PDS Imaging Node and the Cassini Imaging Team.  Here are some highlights:
The label (LBL) files supplied by the Cassini Project have four different ways of identifying the target, none of which is foolproof.
TARGET_NAME encodes the reference object that was used in the pointing design.  Since all Rings observations are implemented by locking onto Saturn with an offset, this parameter is “Saturn” for Rings observations.  This problem has long been known, so next was developed:
TARGET_DESC, which encodes the brightness model that was used to calculate the exposure.  This does a much better job than TARGET_NAME, but still may not be foolproof.
TARGET_LIST contains all the objects that appear in the image, as calculated by the Imaging Team’s Autonav software.  This duplicates our Surface Geometry Target functionality, which does a fine job, so we can ignore it.  Like Surface Geometry Target, TARGET_LIST does not detect rings.  There is supposedly a RING_LIST that compensates for this deficiency, but I do not see it in the LBL file (it may have been separately provided by the Imaging Team to the Imaging Node).
The fourth and fifth characters of OBSERVATION_ID are a two-letter cipher that encode the observer’s intended target.  Most Rings observations are RI, though there are also codes for specific rings (most commonly used was RF, as Carl Murray wanted to distinguish his observations as not belonging to the Team Leader).
Ben Knowles pointed out that TARGET_NAME does retain some usefulness because it encodes some targets that are not in PCK kernels, including “a handful of specific star targets (VEGA, FOMALHAUT, SPICA), as well as the asteroid MASURSKY. There may be others as well.”
I’m pretty sure we’ve discussed this before.  Actually, consulting my archives, it looks like we had a thread in March and another in April.
Looking at the OPUS entry you linked Rob, it appears that “Cassini Target Code” represents the two-letter code from OBSERVATION_ID, which is great.  I also see that “Target Description” is set to Saturn-Rings, which perhaps is a representation of TARGET_DESC?  This also seems great.  The one problem is “Intended Target Name,” which, as you point out, is set to Saturn.  There is a corresponding “Nominal Target Class” which is set to Planet.  If this is a representation of TARGET_NAME, then it needs to be fixed.  My records indicate that we historically would use TARGET_NAME but would override it when the two-letter code indicated a Rings observation.  What goes into Intended Target Name at this time?
On Dec 20, 2018, at 2:09 PM, Robert French <notifications<EMAIL_ADDRESS>wrote:
This search (from the API result count test):
https://tools.pds-rings.seti.org/#/instrument=Cassini+ISS&surfacegeometrytargetname=SATURN&SURFACEGEOsaturnsubobserverplanetographiclatitude1=50&SURFACEGEOsaturncenterresolution1=5&SURFACEGEOsaturncenterresolution2=10&target=SATURN&view=detail&browse=gallery&colls_browse=gallery&page=1&gallery_data_viewer=true&limit=100&order=time1&cols=opusid,instrument,planet,target,time1,observationduration&widgets=instrument,surfacegeometrytargetname,SURFACEGEOsaturnsubobserverplanetographiclatitude1,SURFACEGEOsaturncenterresolution1,target&widgets2=&detail=co-iss-n1545019067
shows that the TARGET is Saturn, but the image is clearly of Saturn Rings. If you look at the label file:
https://pds-rings.seti.org/holdings/volumes/COISS_2xxx/COISS_2027/data/1545009755_1545083950/N1545019067_1.LBL
you'll see that TARGET_DESC is "Saturn-Rings".
Perhaps in some cases other than "Sky" we should let TARGET_DESC override TARGET?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/SETI/pds-opus/issues/533, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AR3CBkWTMriZsBoNapmjzY3Ko-w46H-nks5u7Ap8gaJpZM4Zc_ml.

Matthew S. Tiscareno, Senior Research Scientist
SETI Institute, 189 Bernardo Avenue #200, Mountain View CA 94043

Yes, we've discussed this at length, but either we didn't get this level of clarity on TARGET_NAME vs. TARGET_DESC or I misunderstood what we were supposed to do. "Cassini Mission/Cassini Target Code" is taken from the Observation Name. "Cassini ISS/Target Description" is taken from the TARGET_DESC field. Those are both fine as is I think.
The trick comes in the computation of Intended Target Name, which is currently the same for all Cassini instruments.
Here is the basic logic:

If TARGET_NAME is "SATURN" or "SKY" and Cassini Target Code is one of the rings, then the Intended Target Name is "Saturn Rings".
If there is a TARGET_DESC (Cassini ISS only) and TARGET_NAME is "SKY" and Cassini Target Code is "SK" (which I think means "Skeleton Request", not "Sky") and TARGET_DESC has the word "Rings" in it, then the Intended Target Name is "Saturn Rings"
Otherwise, just use whatever is in TARGET_NAME

For the particular image quoted above (there are quite a few in that set):

TARGET_NAME is SATURN
Cassini Target Code is OTHER
TARGET_DESC is Saturn-Rings

So it appears the logic is falling apart because the Target Code is OTHER instead of SKY.
So for Cassini ISS, should TARGET_DESC always override TARGET_NAME if it's something that we can decode?
Perhaps I need to produce a table of all possible TARGET_NAME and TARGET_DESC combinations for us to look at.

I’m not sure whether TARGET_DESC should always override TARGET_NAME.  It would be interesting to see a full list of the combinations that occur, as you mentioned.
This case arises because the current logic does not account for cases where Cassini Target Code is OTHER.  The observation you linked was 035OT_RETARGMR, which was a “medium-resolution retargettable” that was inserted into the observation sequence for the purpose of providing the option to decide at the last minute what the target would be.  In this case, the Imaging Team appears to have decided to target the rings, which is why TARGET_NAME and TARGET_DESC are what they are.
I think you could insert “OT” alongside “SK” in your second rule, and it would at least take care of cases like the one you’ve linked.
Here is a search of all images for which Cassini Target Code is OT and Intended Target Name is Saturn:
https://tools.pds-rings.seti.org/opus/#/instrument=Cassini+ISS&CASSINIobsname=&CASSINItargetcode=OT+(Other)&target=Saturn
Clearly, very few of these are images of Saturn.  I think most of them are for calibration.
Incidentally, I really like the way the Cassini Target Code widget has checkboxes, so that the user can only select values that actually occur.  Might Target Description have the same form?  Doesn’t it have a relatively limited number of occurring values?  I would be interested to see green-number results there for the query I just linked.
On Dec 20, 2018, at 3:08 PM, Robert French <notifications<EMAIL_ADDRESS>wrote:
Yes, we've discussed this at length, but either we didn't get this level of clarity on TARGET_NAME vs. TARGET_DESC or I misunderstood what we were supposed to do. "Cassini Mission/Cassini Target Code" is taken from the Observation Name. "Cassini ISS/Target Description" is taken from the TARGET_DESC field. Those are both fine as is I think.
The trick comes in the computation of Intended Target Name, which is currently the same for all Cassini instruments.
Here is the basic logic:

If TARGET_NAME is "SATURN" or "SKY" and Cassini Target Code is one of the rings, then the Intended Target Name is "Saturn Rings".
If there is a TARGET_DESC (Cassini ISS only) and TARGET_NAME is "SKY" and Cassini Target Code is "SK" (which I think means "Skeleton Request", not "Sky") and TARGET_DESC has the word "Rings" in it, then the Intended Target Name is "Saturn Rings"
Otherwise, just use whatever is in TARGET_NAME

For the particular image quoted above (there are quite a few in that set):

TARGET_NAME is SATURN
Cassini Target Code is OTHER
TARGET_DESC is Saturn-Rings

So it appears the logic is falling apart because the Target Code is OTHER instead of SKY.
So for Cassini ISS, should TARGET_DESC always override TARGET_NAME if it's something that we can decode?
Perhaps I need to produce a table of all possible TARGET_NAME and TARGET_DESC combinations for us to look at.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/SETI/pds-opus/issues/533#issuecomment-449165779, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AR3CBqIjxnCnJ5uypdUnM0m_gWk9iUC0ks5u7BhlgaJpZM4Zc_ml.

Matthew S. Tiscareno, Senior Research Scientist
SETI Institute, 189 Bernardo Avenue #200, Mountain View CA 94043

Here is the complete list of image that have a target code of (RI, RA, RB...) or a target description that includes "ring". Fields are target_desc, target_name, target_code.
iss.txt

Here are all the target_descs in COISS. We certainly could also make this a checkbox field.
ATLAS
CALYPSO
CONSTANT I/F
DARK
DIONE
DIONE, RHEA, MIMAS(?), RINGS
ENCELADUS
ENCELADUS, RINGS
EPIMETHEUS
F RING
FLIGHT SOFTWARE TEST
GENERIC-SATELLITE
HELENE
HYPERION
I/F
IAPETUS
IAPETUS FP1
JANUS
JUPITER
METHON
MIMAS
PANDORA
PHOEBE
PLUTO
POLYDEUCES
PROBE
PROMETHEUS
RHEA
ROCK
ROCKS
SATELLITE SEARCH
SATURN
SATURN-DRING
SATURN-ERING
SATURN-FRING
SATURN-GRING
SATURN-RINGS
STAR
STAR -- CW LEO
STAR -- ETA CAR
TELESTO
TETHYS
TETHYS, RHEA, RINGS
TITAN
TITAN CB3 MOVIE AND SI
TITAN LIMB

TARGET_DESC contains "RING" and Intended Target not "S RINGS":
+---------------------+-----------------+----------+
| cassini_target_code | Intended Target | count(*) |
+---------------------+-----------------+----------+
| NULL                | SATURN          |      267 |
| AG                  | AEGAEON         |       46 |
| AN                  | ANTHE           |       26 |
| CO                  | UNKNOWN         |       21 |
| DI                  | DIONE           |        3 |
| DR                  | UNKNOWN         |       10 |
| EN                  | ENCELADUS       |      179 |
| EN                  | SATURN          |        5 |
| IC                  | SUN             |       74 |
| IC                  | UNKNOWN         |       63 |
| MI                  | MIMAS           |       13 |
| OT                  | ANTHE           |       31 |
| OT                  | ATLAS           |       16 |
| OT                  | DAPHNIS         |       83 |
| OT                  | LOGE            |        2 |
| OT                  | METHONE         |       48 |
| OT                  | PALLENE         |       30 |
| OT                  | PAN             |      143 |
| OT                  | PROMETHEUS      |        5 |
| OT                  | SATURN          |     2849 |
| OT                  | SKY             |       36 |
| PM                  | SATURN          |       46 |
| PN                  | PAN             |      177 |
| RE                  | ENCELADUS       |       14 |
| RE                  | METHONE         |        5 |
| RE                  | PALLENE         |        5 |
| RF                  | PROMETHEUS      |     1895 |
| RG                  | AEGAEON         |       26 |
| RH                  | EPIMETHEUS      |        8 |
| RH                  | SATURN          |       26 |
| RI                  | AEGAEON         |      541 |
| RI                  | ANTHE           |      174 |
| RI                  | ATLAS           |      109 |
| RI                  | CALYPSO         |        4 |
| RI                  | DAPHNIS         |      241 |
| RI                  | EARTH           |       13 |
| RI                  | ENCELADUS       |       10 |
| RI                  | EPIMETHEUS      |        3 |
| RI                  | METHONE         |       99 |
| RI                  | PALLENE         |       62 |
| RI                  | PAN             |      119 |
| RI                  | PANDORA         |       20 |
| RI                  | SUN             |       16 |
| RI                  | UNKNOWN         |      666 |
| SA                  | ENCELADUS       |        2 |
| SA                  | SATURN          |     1004 |
| SA                  | SKY             |       12 |
| SK                  | ATLAS           |        6 |
| SK                  | JANUS           |        4 |
| SK                  | PAN             |        5 |
| SR                  | UNKNOWN         |       24 |
| ST                  | SKY             |      795 |
| TE                  | TETHYS          |        3 |
| TI                  | TITAN           |        9 |
+---------------------+-----------------+----------+

Limiting to Intended Target "SATURN" and including the TARGET_DESC:
+---------------------+--------------+-----------------+----------+
| cassini_target_code | target_desc  | Intended Target | count(*) |
+---------------------+--------------+-----------------+----------+
| NULL                | Saturn-Rings | SATURN          |      267 |
| EN                  | Saturn-Rings | SATURN          |        5 |
| OT                  | Saturn-Dring | SATURN          |       44 |
| OT                  | Saturn-Fring | SATURN          |       94 |
| OT                  | Saturn-Gring | SATURN          |      436 |
| OT                  | Saturn-Rings | SATURN          |     2275 |
| PM                  | Saturn-Rings | SATURN          |       46 |
| RH                  | Saturn-Rings | SATURN          |       26 |
| SA                  | Saturn-Fring | SATURN          |      102 |
| SA                  | Saturn-Gring | SATURN          |       12 |
| SA                  | Saturn-Rings | SATURN          |      890 |
+---------------------+--------------+-----------------+----------+

Clearly it is reasonable to use TARGET="SATURN" and CODE="OT" and DESC contains "RING" as a trigger for "S RINGS".
The 5 EN images are edge-on rings with Enceladus nearby.
The 46 PM images don't actually seem to have Prometheus in them, just rings.
The 26 RH images just have the rings.
The 267 NULL images just have the rings.
And, frankly, the 1000 SA images mainly have the rings, although Saturn might be a little bit in the FOV:
https://tools.pds-rings.seti.org/opus/#/instrument=Cassini+ISS&COISStargetdesc=ring&target=Saturn&planet=Saturn&CASSINItargetcode=SA+(Saturn)&view=browse&browse=gallery&colls_browse=gallery&page=1&gallery_data_viewer=true&limit=100&order=time1&cols=opusid,instrument,planet,target,time1,observationduration&widgets=planet,target,COISStargetdesc,CASSINItargetcode,instrument&widgets2=
Given all of this, I think it would not be unreasonable to say that if TARGET_NAME is SATURN and TARGET_DESC contains "RING" then we always set the Intended Target to "S RINGS" regardless of the Target Code.
Now, looking at current Intended Target of "SKY":
+---------------------+--------------+-----------------+----------+
| cassini_target_code | target_desc  | Intended Target | count(*) |
+---------------------+--------------+-----------------+----------+
| OT                  | Saturn-Rings | SKY             |       36 |
| SA                  | Saturn-Rings | SKY             |       12 |
| ST                  | Saturn-Dring | SKY             |       28 |
| ST                  | Saturn-Fring | SKY             |       51 |
| ST                  | Saturn-Rings | SKY             |      716 |
+---------------------+--------------+-----------------+----------+

Looking at all of these images:
The "ST" ones really do have a star in them, usually shining through the rings.
The "SA" ones have just rings.
The "OT" ones just rings.
_So, assuming we want to keep the "ST" ones as "STAR", the final logic is:
If TARGET_NAME == 'SATURN' and TARGET_DESC contains "Ring" and TARGET_CODE != 'ST':
Intended Target Name = "S RINGS"_

These are fun.  I think a checkbox field would be nice.
I wonder whether we want to keep them all verbatim, or translate some of the more awkward ones.  For example, “DIONE, RHEA, MIMAS(?), RINGS” is used for a grand total of three images, all from a Rev 004 OPNAV that does indeed include the four named bodies all in one image.  I wonder whether this, along with the other entries containing commas as well as GENERIC-SATELLITE and SATELLITE SEARCH, could be combined into an entry named something like “Generic or Multiple Icy Satellites.”
Another example might be to subsume “IAPETUS FP1” (used for only 2 images in 2015) into “IAPETUS,” and the like.
On Dec 20, 2018, at 6:04 PM, Robert French <notifications<EMAIL_ADDRESS>wrote:
Here are all the target_descs in COISS. We certainly could also make this a checkbox field.

Reply to this email directly, view it on GitHubhttps://github.com/SETI/pds-opus/issues/533#issuecomment-449215025, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AR3CBr7jufLFc9it2sNghcGmbK4cYDJDks5u7EGhgaJpZM4Zc_ml.

Matthew S. Tiscareno, Senior Research Scientist
SETI Institute, 189 Bernardo Avenue #200, Mountain View CA 94043

Your list doesn’t have an entry for target_desc = Saturn, target_name = Saturn, and target_code = SA, and yet OPUS indicates that there are nearly 100,000 such images
https://tools.pds-rings.seti.org/opus/#/instrument=Cassini+ISS&COISStargetdesc=SATURN&CASSINItargetcode=SA+(Saturn)&target=Saturn
Do you know why this is?
On Dec 20, 2018, at 5:14 PM, Robert French <notifications<EMAIL_ADDRESS>wrote:
Here is the complete list of image that have a target code of (RI, RA, RB...) or a target description that includes "ring". Fields are target_desc, target_name, target_code.
iss.txthttps://github.com/SETI/pds-opus/files/2701139/iss.txt
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/SETI/pds-opus/issues/533#issuecomment-449199616, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AR3CBuFIh5kaUvgPGcDv_8hrFFTafGu3ks5u7DXtgaJpZM4Zc_ml.

Matthew S. Tiscareno, Senior Research Scientist
SETI Institute, 189 Bernardo Avenue #200, Mountain View CA 94043

The 46 images with target_code = Prometheus but target_desc = Saturn-Rings are the infamous PROMPROP, in which we wanted to image Prometheus twice with propeller observations in between, and it would have been operationally more complicated to make them separate observations in an administrative sense.
The 26 images with target_code = Rhea but target_desc = Saturn-Rings had been sequenced as Rhea Ring-Plane Crossing observation time to search for rings around Rhea, except that the phase angle was too high for part of the observation time, so that time was repurposed to image the main rings but without changing the administrative booking of the observation.
The observations with target_code = Star are times when ISS was “riding” on VIMS or UVIS occultations, providing context images that would help interpret the occultations.  There’s no reason why these couldn’t be atmospheric occultations, but looking at the images in the gallery it seems to me that every one of them is a rings occultation.  There are a few images with the star in an empty field, but if you look closely those are immediately followed by images of the same star going behind the rings.  Thus, you could arguably set these also to S RINGS, though STAR is also a fine choice.
All this to say that I generally agree with your conclusions, Rob.
On Dec 20, 2018, at 7:04 PM, Robert French <notifications<EMAIL_ADDRESS>wrote:
TARGET_DESC contains "RING" and Intended Target not "S RINGS":
+---------------------+-----------------+----------+
| cassini_target_code | Intended Target | count(*) |
+---------------------+-----------------+----------+
| NULL                | SATURN          |      267 |
| AG                  | AEGAEON         |       46 |
| AN                  | ANTHE           |       26 |
| CO                  | UNKNOWN         |       21 |
| DI                  | DIONE           |        3 |
| DR                  | UNKNOWN         |       10 |
| EN                  | ENCELADUS       |      179 |
| EN                  | SATURN          |        5 |
| IC                  | SUN             |       74 |
| IC                  | UNKNOWN         |       63 |
| MI                  | MIMAS           |       13 |
| OT                  | ANTHE           |       31 |
| OT                  | ATLAS           |       16 |
| OT                  | DAPHNIS         |       83 |
| OT                  | LOGE            |        2 |
| OT                  | METHONE         |       48 |
| OT                  | PALLENE         |       30 |
| OT                  | PAN             |      143 |
| OT                  | PROMETHEUS      |        5 |
| OT                  | SATURN          |     2849 |
| OT                  | SKY             |       36 |
| PM                  | SATURN          |       46 |
| PN                  | PAN             |      177 |
| RE                  | ENCELADUS       |       14 |
| RE                  | METHONE         |        5 |
| RE                  | PALLENE         |        5 |
| RF                  | PROMETHEUS      |     1895 |
| RG                  | AEGAEON         |       26 |
| RH                  | EPIMETHEUS      |        8 |
| RH                  | SATURN          |       26 |
| RI                  | AEGAEON         |      541 |
| RI                  | ANTHE           |      174 |
| RI                  | ATLAS           |      109 |
| RI                  | CALYPSO         |        4 |
| RI                  | DAPHNIS         |      241 |
| RI                  | EARTH           |       13 |
| RI                  | ENCELADUS       |       10 |
| RI                  | EPIMETHEUS      |        3 |
| RI                  | METHONE         |       99 |
| RI                  | PALLENE         |       62 |
| RI                  | PAN             |      119 |
| RI                  | PANDORA         |       20 |
| RI                  | SUN             |       16 |
| RI                  | UNKNOWN         |      666 |
| SA                  | ENCELADUS       |        2 |
| SA                  | SATURN          |     1004 |
| SA                  | SKY             |       12 |
| SK                  | ATLAS           |        6 |
| SK                  | JANUS           |        4 |
| SK                  | PAN             |        5 |
| SR                  | UNKNOWN         |       24 |
| ST                  | SKY             |      795 |
| TE                  | TETHYS          |        3 |
| TI                  | TITAN           |        9 |
+---------------------+-----------------+----------+
Limiting to Intended Target "SATURN" and including the TARGET_DESC:
+---------------------+--------------+-----------------+----------+
| cassini_target_code | target_desc  | Intended Target | count(*) |
+---------------------+--------------+-----------------+----------+
| NULL                | Saturn-Rings | SATURN          |      267 |
| EN                  | Saturn-Rings | SATURN          |        5 |
| OT                  | Saturn-Dring | SATURN          |       44 |
| OT                  | Saturn-Fring | SATURN          |       94 |
| OT                  | Saturn-Gring | SATURN          |      436 |
| OT                  | Saturn-Rings | SATURN          |     2275 |
| PM                  | Saturn-Rings | SATURN          |       46 |
| RH                  | Saturn-Rings | SATURN          |       26 |
| SA                  | Saturn-Fring | SATURN          |      102 |
| SA                  | Saturn-Gring | SATURN          |       12 |
| SA                  | Saturn-Rings | SATURN          |      890 |
+---------------------+--------------+-----------------+----------+
Clearly it is reasonable to use TARGET="SATURN" and CODE="OT" and DESC contains "RING" as a trigger for "S RINGS".
The 5 EN images are edge-on rings with Enceladus nearby.
The 46 PM images don't actually seem to have Prometheus in them, just rings.
The 26 RH images just have the rings.
The 267 NULL images just have the rings.
And, frankly, the 1000 SA images mainly have the rings, although Saturn might be a little bit in the FOV:
https://tools.pds-rings.seti.org/opus/#/instrument=Cassini+ISS&COISStargetdesc=ring&target=Saturn&planet=Saturn&CASSINItargetcode=SA+(Saturn)&view=browse&browse=gallery&colls_browse=gallery&page=1&gallery_data_viewer=true&limit=100&order=time1&cols=opusid,instrument,planet,target,time1,observationduration&widgets=planet,target,COISStargetdesc,CASSINItargetcode,instrument&widgets2=
Given all of this, I think it would not be unreasonable to say that if TARGET_NAME is SATURN and TARGET_DESC contains "RING" then we always set the Intended Target to "S RINGS" regardless of the Target Code.
Now, looking at current Intended Target of "SKY":
+---------------------+--------------+-----------------+----------+
| cassini_target_code | target_desc  | Intended Target | count(*) |
+---------------------+--------------+-----------------+----------+
| OT                  | Saturn-Rings | SKY             |       36 |
| SA                  | Saturn-Rings | SKY             |       12 |
| ST                  | Saturn-Dring | SKY             |       28 |
| ST                  | Saturn-Fring | SKY             |       51 |
| ST                  | Saturn-Rings | SKY             |      716 |
+---------------------+--------------+-----------------+----------+
Looking at all of these images:
The "ST" ones really do have a star in them, usually shining through the rings.
The "SA" ones have just rings.
The "OT" ones just rings.
_So, assuming we want to keep the "ST" ones as "STAR", the final logic is:
If TARGET_NAME == 'SATURN' and TARGET_DESC contains "Ring" and TARGET_CODE != 'ST':
Intended Target Name = "S RINGS"_
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/SETI/pds-opus/issues/533#issuecomment-449225848, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AR3CBsNGRzjhq9oo5wl7s_rMAV_HRNKRks5u7E-kgaJpZM4Zc_ml.

Matthew S. Tiscareno, Senior Research Scientist
SETI Institute, 189 Bernardo Avenue #200, Mountain View CA 94043

I would want to clean them up a little bit. I suggest:
DIONE, RHEA, MIMAS(?), RINGS & GENERIC-SATELLITE & SATELLITE SEARCH & TETHYS, RHEA, RINGS => ICY SATELLITES
ENCELADUS, RINGS => ENCELADUS
IAPETUS FP1 => IAPETUS
METHON => METHONE
ROCK => ROCKS
STAR -- CW LEO & STAR -- ETA CAR => STAR (There are only 2 each)
12/20/2018 11:13 PM, Matthew Tiscareno wrote:
These are fun. I think a checkbox field would be nice.
I wonder whether we want to keep them all verbatim, or translate some of the more awkward ones. For example, “DIONE, RHEA, MIMAS(?), RINGS” is used for a grand total of three images, all from a Rev 004 OPNAV that does indeed include the four named bodies all in one image. I wonder whether this, along with the other entries containing commas as well as GENERIC-SATELLITE and SATELLITE SEARCH, could be combined into an entry named something like “Generic or Multiple Icy Satellites.”
Another example might be to subsume “IAPETUS FP1” (used for only 2 images in 2015) into “IAPETUS,” and the like.
On Dec 20, 2018, at 6:04 PM, Robert French <notifications@github.commailto:notifications@github.commailto:notifications<EMAIL_ADDRESS>wrote:
Here are all the target_descs in COISS. We certainly could also make this a checkbox field.

Reply to this email directly, view it on GitHubhttps://github.com/SETI/pds-opus/issues/533#issuecomment-449215025https://github.com/SETI/pds-opus/issues/533#issuecomment-449215025, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AR3CBr7jufLFc9it2sNghcGmbK4cYDJDks5u7EGhgaJpZM4Zc_mlhttps://github.com/notifications/unsubscribe-auth/AR3CBr7jufLFc9it2sNghcGmbK4cYDJDks5u7EGhgaJpZM4Zc_ml.

Matthew S. Tiscareno, Senior Research Scientist
SETI Institute, 189 Bernardo Avenue #200, Mountain View CA 94043
—

As mentioned below these were only results with Target Code (RI, RA, RB...) or with TARGET_DESC containing "RINGS" (i.e. observations we're likely concerned about in this thread). Your example does not match either of those criteria.
On 12/20/2018 11:21 PM, Matthew Tiscareno wrote:
Your list doesn’t have an entry for target_desc = Saturn, target_name = Saturn, and target_code = SA, and yet OPUS indicates that there are nearly 100,000 such images
https://tools.pds-rings.seti.org/opus/#/instrument=Cassini+ISS&COISStargetdesc=SATURN&CASSINItargetcode=SA+(Saturn)&target=Saturn
Do you know why this is?
On Dec 20, 2018, at 5:14 PM, Robert French <notifications@github.commailto:notifications@github.commailto:notifications<EMAIL_ADDRESS>wrote:
Here is the complete list of image that have a target code of (RI, RA, RB...) or a target description that includes "ring". Fields are target_desc, target_name, target_code.
iss.txthttps://github.com/SETI/pds-opus/files/2701139/iss.txthttps://github.com/SETI/pds-opus/files/2701139/iss.txt
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/SETI/pds-opus/issues/533#issuecomment-449199616https://github.com/SETI/pds-opus/issues/533#issuecomment-449199616, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AR3CBuFIh5kaUvgPGcDv_8hrFFTafGu3ks5u7DXtgaJpZM4Zc_mlhttps://github.com/notifications/unsubscribe-auth/AR3CBuFIh5kaUvgPGcDv_8hrFFTafGu3ks5u7DXtgaJpZM4Zc_ml.

Matthew S. Tiscareno, Senior Research Scientist
SETI Institute, 189 Bernardo Avenue #200, Mountain View CA 94043

Seems good to me.
On Dec 21, 2018, at 11:06 AM, Robert French <notifications<EMAIL_ADDRESS>wrote:
I would want to clean them up a little bit. I suggest:
DIONE, RHEA, MIMAS(?), RINGS & GENERIC-SATELLITE & SATELLITE SEARCH & TETHYS, RHEA, RINGS => ICY SATELLITES
ENCELADUS, RINGS => ENCELADUS
IAPETUS FP1 => IAPETUS
METHON => METHONE
ROCK => ROCKS
STAR -- CW LEO & STAR -- ETA CAR => STAR (There are only 2 each)
12/20/2018 11:13 PM, Matthew Tiscareno wrote:
These are fun. I think a checkbox field would be nice.
I wonder whether we want to keep them all verbatim, or translate some of the more awkward ones. For example, “DIONE, RHEA, MIMAS(?), RINGS” is used for a grand total of three images, all from a Rev 004 OPNAV that does indeed include the four named bodies all in one image. I wonder whether this, along with the other entries containing commas as well as GENERIC-SATELLITE and SATELLITE SEARCH, could be combined into an entry named something like “Generic or Multiple Icy Satellites.”
Another example might be to subsume “IAPETUS FP1” (used for only 2 images in 2015) into “IAPETUS,” and the like.
On Dec 20, 2018, at 6:04 PM, Robert French <notifications@github.commailto:notifications@github.commailto:notifications@github.commailto:notifications<EMAIL_ADDRESS>wrote:
Here are all the target_descs in COISS. We certainly could also make this a checkbox field.
ATLAS
CALYPSO
CONSTANT I/F
DARK
DIONE
DIONE, RHEA, MIMAS(?), RINGS
ENCELADUS
ENCELADUS, RINGS
EPIMETHEUS
F RING
FLIGHT SOFTWARE TEST
GENERIC-SATELLITE
HELENE
HYPERION
I/F
IAPETUS
IAPETUS FP1
JANUS
JUPITER
METHON
MIMAS
PANDORA
PHOEBE
PLUTO
POLYDEUCES
PROBE
PROMETHEUS
RHEA
ROCK
ROCKS
SATELLITE SEARCH
SATURN
SATURN-DRING
SATURN-ERING
SATURN-FRING
SATURN-GRING
SATURN-RINGS
STAR
STAR -- CW LEO
STAR -- ETA CAR
TELESTO
TETHYS
TETHYS, RHEA, RINGS
TITAN
TITAN CB3 MOVIE AND SI
TITAN LIMB

Matthew S. Tiscareno, Senior Research Scientist
SETI Institute, 189 Bernardo Avenue #200, Mountain View CA 94043

Okay, I’m sorry that my previous question was based on misunderstanding.
This is a useful reference, and I’d like to flag it for further study.  I might sometime go through and see what the images with each combination actually look like.  The question is whether our logic for determining the Intended Target derives the correct value.
On Dec 20, 2018, at 5:14 PM, Robert French <notifications<EMAIL_ADDRESS>wrote:
Here is the complete list of image that have a target code of (RI, RA, RB...) or a target description that includes "ring". Fields are target_desc, target_name, target_code.
iss.txthttps://github.com/SETI/pds-opus/files/2701139/iss.txt

Matthew S. Tiscareno, Senior Research Scientist
SETI Institute, 189 Bernardo Avenue #200, Mountain View CA 94043

It’s actually a little tricky to search for these combinations in OPUS because we don’t store the original TARGET_NAME, only our derived intended target name. Do you think we should store the original version in the Cassini Mission section?

I know that the Imaging Node does, and there are reasons why advanced users might want to search on this parameter.
It might be good to do it but with a disclaimer in the description that this is an “under the hood” feature that is intended only for advanced users who understand what the keyword really means.
I don’t know how Mark feels about this.
On Dec 22, 2018, at 7:48 AM, Robert French <notifications<EMAIL_ADDRESS>wrote:
It’s actually a little tricky to search for these combinations in OPUS because we don’t store the original TARGET_NAME, only our derived intended target name. Do you think we should store the original version in the Cassini Mission section?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/SETI/pds-opus/issues/533#issuecomment-449579224, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AR3CBrR7-xXSSZhSGE_O2h9dJjEa7MBQks5u7lQ-gaJpZM4Zc_ml.

Matthew S. Tiscareno, Senior Research Scientist
SETI Institute, 189 Bernardo Avenue #200, Mountain View CA 94043

OK I will go ahead and make the two primary changes:

Make Cassini "Target Description" a checkbox field, with appropriate mappings to clean things up.
Change the logic for detecting S Rings to be:

    If ((TARGET_NAME == 'SATURN' or TARGET_NAME == 'SKY') and
         TARGET_DESC contains "Ring" and TARGET_CODE != 'ST'):
        Intended Target Name = "S RINGS"

I will leave the addition of an "Original Target Name" field until we talk to Mark. To date we have avoided adding search fields that are useful only for "advanced" users, relegating those to the Imaging Node instead, and keeping OPUS more for the general end user.

Here is the complete list of all target combinations, along with the number of observations for each. Note I'm using the COISS_2116 cumulative index, which Michael Aye has noted is missing COISS_2115 entirely, so it's possible some combination is missing from this.
iss.txt

I have implemented TARGET_DESC (cleaned up) as a checkbox field, and changed the derivation of "Saturn Rings". I'm going to close this and open a new issue about whether we want to have an "original TARGET_NAME" constraint.

And here is the same query before the change:
+-------------+---------------------+------------------------------+----------+
| target_name | cassini_target_code | target_desc                  | count(*) |
+-------------+---------------------+------------------------------+----------+
| S RINGS     | RA                  | Saturn-Rings                 |      452 |
| S RINGS     | RA                  | STAR                         |        6 |
| S RINGS     | RB                  | Constant I/F                 |        8 |
| S RINGS     | RB                  | Saturn                       |        8 |
| S RINGS     | RB                  | Saturn-Rings                 |      868 |
| S RINGS     | RB                  | STAR                         |        6 |
| S RINGS     | RC                  | Saturn                       |        3 |
| S RINGS     | RC                  | Saturn-Fring                 |        6 |
| S RINGS     | RC                  | Saturn-Rings                 |      742 |
| S RINGS     | RD                  | Saturn-Dring                 |       52 |
| S RINGS     | RD                  | Saturn-Fring                 |       43 |
| S RINGS     | RD                  | Saturn-Rings                 |       52 |
| S RINGS     | RE                  | Saturn                       |        1 |
| S RINGS     | RE                  | Saturn-Dring                 |       13 |
| S RINGS     | RE                  | Saturn-Ering                 |     2651 |
| S RINGS     | RE                  | Saturn-Fring                 |      175 |
| S RINGS     | RE                  | Saturn-Rings                 |      321 |
| S RINGS     | RF                  | F Ring                       |     1470 |
| S RINGS     | RF                  | Prometheus                   |        3 |
| S RINGS     | RF                  | Saturn                       |        3 |
| S RINGS     | RF                  | Saturn-Fring                 |     1669 |
| S RINGS     | RF                  | Saturn-Rings                 |    13627 |
| S RINGS     | RG                  | Saturn-Ering                 |       33 |
| S RINGS     | RG                  | Saturn-Gring                 |      852 |
| S RINGS     | RI                  | Atlas                        |        2 |
| S RINGS     | RI                  | Constant I/F                 |      369 |
| S RINGS     | RI                  | Epimetheus                   |        3 |
| S RINGS     | RI                  | F Ring                       |      176 |
| S RINGS     | RI                  | i/f                          |      178 |
| S RINGS     | RI                  | Janus                        |        2 |
| S RINGS     | RI                  | Mimas                        |       12 |
| S RINGS     | RI                  | Prometheus                   |        1 |
| S RINGS     | RI                  | Rhea                         |        5 |
| S RINGS     | RI                  | Rocks                        |        4 |
| S RINGS     | RI                  | Satellite Search             |       48 |
| S RINGS     | RI                  | Saturn                       |      885 |
| S RINGS     | RI                  | Saturn-Dring                 |     1023 |
| S RINGS     | RI                  | Saturn-Ering                 |     3109 |
| S RINGS     | RI                  | Saturn-Fring                 |     3307 |
| S RINGS     | RI                  | Saturn-Gring                 |     1380 |
| S RINGS     | RI                  | Saturn-Rings                 |    75476 |
| S RINGS     | RI                  | STAR                         |        4 |
| S RINGS     | RI                  | Tethys                       |        2 |
| S RINGS     | RI                  | Titan                        |        6 |
| S RINGS     | SK                  | Dione, Rhea, Mimas(?), Rings |        3 |
| S RINGS     | SK                  | Enceladus, Rings             |        3 |
| S RINGS     | SK                  | Saturn-Rings                 |       93 |
| S RINGS     | SK                  | Tethys, Rhea, Rings          |        3 |
+-------------+---------------------+------------------------------+----------+

For TARGET_DESC, I'm also going to combine "F Ring" in with "Saturn-Fring". They appear to be the same kind of observation.

And I'm mapping "--" and "UNK" to "UNKNOWN".


Initial configuration

Addition of basic helm chart files.
Files added:

Statefulset.yaml - This file will have the config related to Kubernetes Statefulset
Headless-service.yaml - This file will have the service related to the database through which we can port forward the endpoint
Persistent Volume.yaml - Creates the Persistent Volume for the deployment -> configures storage and volume settings
PVC - Persistent Volume Claim for the Persistent Volume

Signed-off-by: Abhijith Ganesh<EMAIL_ADDRESS>

The thing is, all Kubernetes charts that run databases are called operators

Only if they really introduce custom resources right? I can find lots of MariaDB, MySQL, PostgreSQL helm charts that have no mention of operators.

Yes , I forgot to add the CRD , this is still work in progress

Only if they really introduce custom resources right? I can find lots of MariaDB, MySQL, PostgreSQL helm charts that have no mention of operators.

I was also under the assumption we need a CRD for Terminus. Don't we?

@rrooij Can you now guide me with what needs to be added?


Having issue on windows

When I install it on windows using pm2 install pm2-server-monit  its went crazy and keep opening new command windows and closing them.  It acted like I have installed some virus.  In a second it will open multiple command prompts and close them automatically and this goes on forever.

This module is made for Unix systems only. Sorry.


Error Fix

IBM Cloud toolchain: Delivery Pipeline deployed products to nyu-devops-products, including commit 6cb8a20c100916c87a5519f33b5b368919a8c11d


Invalid characters in Title

Description
When I run the Stop and Start actions in the ZedControl web interface for the AlpineDocker application from the App Marketplace, I get an error message:
"Deactivate Request to Edge App Instance 'kora013' failed: Invalid characters in Title"
but the title "test application 013" was taken when creating the application without any comment.
Steps to Reproduce
Go to application management interface:
https://zedcontrol.zededa.net/edge-app-instances/393c7139-118a-4667-97dc-9742c70ef935/details/status
and try to choose the “Stop” or “Start” menu items.
Actual Results
Error message "Invalid characters in Title" without changing the state and status of the application.
Expected Results
The application must be stopped and started.
Additional Information

With title "test application 014" I get the same message error -- "Invalid characters in Title". But with title "test app 015" -- without errors, and after changed  title to "test application 015" -- also without errors.

Github issues in this repo are only for the EVE project (open-source) - please contact Zededa for the support regarding the commercial controller.


How to get the epic JS with no encryption？

Now the js in epic website is encrypted,  Could you tell me how you get the unencrypted version?

The site isn't encrypted. Most of it is uglified using webpack. Just formatting it is the best way to understand it.
The Talon-related code is further obfuscated because Epic didn't like me reverse engineering it. I was able to copy it before they added the obfuscation in talon-harness.ts. Otherwise, the best way to understand obfuscated code is to throw in some breakpoints and debug.


Release language-detection v1.7.0-amd64-preview

@shiwenmichael : Thanks for your contribution! The author(s) have been notified to review your proposed change.


play with idea it's a nagios alternative in the readme wording

base it on the following nagios citation -
http://nagios.sourceforge.net/docs/3_0/beginners.html
Relax - it's going to take some time. Don't expect to be able to get things working exactly the way you want them right off the bat. it's not that easy. Setting up Nagios can involve a bit of work - partly because of the options that Nagios offers, partly because you need to know what to monitor on your network (and how best to do it).
It takes several seconds in our case + we have equal intelligence and work with nagios our of the box

this is for much later on


docs: Improve the video playback guide

Improved the video playback guide.
This is the first draft of this guide, please let me know your thoughts/suggestions.

@victorges @adamsoffer Can you please the Create a playback URL section? Just want to confirm if it is correct as I don't have much information regarding the playback URL.
Also, please let me know if I need to add new sections or improve other sections.

✅  Moved the Livepeer client outside of the index function
✅  Added embeddable player
✅  Added @livepeer/react and livepeer to package.json
✅  Replaced video with HLS Player
and a few more wording changes to the guides


Calendar functionality not working since BST change

Version of the custom_component
1.7.2
Describe the bug
Since the clocks went forward last night (maybe a coincidence but it was working yesterday), the calendar functionality of the integration is no longer working. There are no calendar entries for the daily charge and the following error shows up on the calendar page:

First thing I did was update to 1.7.2 because there was a fix relating to timezones, but that doesn't seem to have resolved the issue.
I'm seeing a recurring error in the logs, which I've pasted below.
Debug log

Logger: aiohttp.server
Source: custom_components/foxess_em/calendar.py:62
Integration: FoxESS - Energy Management ([documentation](https://github.com/nathanmarlor/foxess_em), [issues](https://github.com/nathanmarlor/foxess_em/issues))
First occurred: 11:35:51 (8 occurrences)
Last logged: 11:44:11

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 67, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 227, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 80, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 234, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 146, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/calendar/__init__.py", line 446, in get
    calendar_event_list = await entity.async_get_events(
  File "/config/custom_components/foxess_em/calendar.py", line 62, in async_get_events
    summary += f"Import: {str(round(values['import'], 2))} / "
KeyError: 'import'

Apologies - I half expected something to break even though I'd been super careful with datetimes. All sorted in #236

I've updated to 1.7.4 and the issue still persists:
Source: custom_components/foxess_em/calendar.py:56
Integration: FoxESS - Energy Management ([documentation](https://github.com/nathanmarlor/foxess_em), [issues](https://github.com/nathanmarlor/foxess_em/issues))
First occurred: 14:59:55 (6 occurrences)
Last logged: 15:01:24

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 67, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 227, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 80, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 234, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 146, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/calendar/__init__.py", line 446, in get
    calendar_event_list = await entity.async_get_events(
  File "/config/custom_components/foxess_em/calendar.py", line 56, in async_get_events
    charge = str(round(values["total"], 2))
KeyError: 'total'```

Hmmpphhh.
Can you try removing the integration and re -adding please?

No problem. I'll probably end up doing it tomorrow, purely because I'm at 8 API hits on Solcast today. Re-installing will take me to the limit.
I'll let you know! Thanks.

No problem. Back to fully functioning on my HA after reloading so I'm sure it'll sort yours too. Hopefully that's the last of the timezone gremlins 😂🤦

All good after a clean reinstallation 👍


fix: update dependencies

Code changes were done with standard --fix.

Coverage remained the same at 98.991% when pulling 557f4ccb761ee2cdd255b4d52df25a850ee80ad3 on AdriVanHoudt:deps into 7907e21c68619b14bffbe549cc80f23f593829b2 on pinojs:master.


test anchor change

Overview

PR Checklist

[ ] Related to designs:
[ ] Related to JIRA ticket: [ABC-123]
[ ] I have run this code to verify it works
[ ] This PR includes unit tests for the code change
[ ] This PR includes testing instructions tests for the code change
[ ] The alpha package of this PR is passing end-to-end tests in all relevant Codecademy repositories

Testing Instructions

Don't make me tap the sign.
PR Links and Envs

Repository
PR Link
PR Env

Monolith
Monolith PR
Monolith Env

Portal
Portal Link
Portal Env

Another Repo
Another Link
Another Env

📬Published Alpha Packages:@codecademy/gamut@55.25.2-alpha.2acb0b.0@codecademy/gamut-kit@0.6.423-alpha.2acb0b.0<EMAIL_ADDRESS>
🚀 Styleguide deploy preview ready!https://668c092e53348d555f08646e--gamut-preview.netlify.appDeploy Logs

🚀 Styleguide deploy preview ready!https://668c184553348d69e00864df--gamut-preview.netlify.appDeploy Logs


Bug with Bitbucket collector after initial commit/pr pull

With one of the recent PRs related to the Bitbucket collector (PR #2678) I've encountered an issue after the collector writes it's first time to the database. It thereafter can't read the time back from the collector_item, and would crash with the exception...
 java.lang.ClassCastException: java.lang.Long cannot be cast to java.util.Date
 	at com.capitalone.dashboard.model.GitRepo.getLastUpdateTime(GitRepo.java:53) ~[bitbucket-scm-collector-3.0.2-SNAPSHOT.jar!/:3.0.4]
 	at com.capitalone.dashboard.collector.GitCollectorTask.collect(GitCollectorTask.java:154) ~[bitbucket-scm-collector-3.0.2-SNAPSHOT.jar!/:3.0.2-SNAPSHOT]
 	at com.capitalone.dashboard.collector.CollectorTask.run(CollectorTask.java:63) ~[core-3.0.4.jar!/:3.0.2-SNAPSHOT]
 	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
 	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) [spring-context-4.2.5.RELEASE.jar!/:4.2.5.RELEASE]
 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212]
 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_212]
 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_212]
 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
 	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212

Did I misconfigure something or is did getLastUpdateTime() miss the refactor of how the collector handles it's time?

@shishirchaturvedi  @rvema @Sbrenthughes

@OneMoreByte I am looking at this.

@Sbrenthughes In my last PR for Bitbucket enhancement, I changed LastUpdateTime field on GitRepo collection from java.util.Date to Long(milliSeconds). It is Hygeia standard to use Time Stamps in long millis. Based on the review comments I've changed 'LastUpdateTime' to long but it is breaking existing users of SCM bitbucket collectors. I can submit a PR to change this field back to java.util.Date. What do you think?
https://github.com/Hygieia/Hygieia/pull/2678/files/c80ecf29b462e618ad01fc81aa5fa1eced5d98cd..ac22374626fc5cc70a72b2eade3773ece2d10397#r252381324

@OneMoreByte You are right. I missed the conversion of LastUpdateTime field (Date->Long). My intention was to use Long for LastDateTime field but somehow missed to update it uniformly.
I have submitted this PR and it would be great if you can validate it.

@OneMoreByte Please let me know if you still encounter any issue. I feel that this change of LastUpdateTime from Date to Long might break existing users fo Bitbucket scm collector. I could be wrong.
cc @Sbrenthughes

@OneMoreByte The PR is in, please validate

@shishirchaturvedi @rvema Sorry to get to this so late on Friday. The PR seemed to do the trick. Thanks so much for your promptness!


Some RemoteApps don't show their Icon in the Taskbar

Describe the bug
When using Remote Apps, some apps like Adobe Illustrator and Photoshop don't have their icons in the Linux host taskbar when launched while others like Microsoft Excel and Word do. This makes application management difficult for me since I can't maximize them if I ever minimize them.
To Reproduce
Steps to reproduce the behavior:

On Windows Guest ungroup programs in the taskbar (I'm not sure if this does anything but I'll mark this anyways)
In the Linux Host terminal run xfreerdp /u:[user] /p:[pass] /app:"C:\Program Files\Adobe\Adobe Photoshop 2021\Photoshop.exe" /v:[ip] to launch Photoshop (or any other problematic application)
The application starts as expected but its icon doesn't show in the Linux Host taskbar

Expected behavior
For the icon to show in the taskbar that allows me to minimize, maximize, etc.
Screenshots
I've attached two screen shots. One with Photoshop which shows the problem I'm trying to describe (the taskbar at the bottom lacks the Photoshop icon) and another with Word which shows the expected behaviour (there's the Word icon in the taskbar)

Application details

FreeRDP version: 2.8.0
Command line used: xfreerdp /u:[user] /p:[pass] /app:"C:\Program Files\Adobe\Adobe Photoshop 2021\Photoshop.exe" /v:[ip]
Output of xfreerdp /buildconfig:

This is FreeRDP version 2.8.0 (2.8.0)
Build configuration: BUILD_TESTING=OFF BUILTIN_CHANNELS=ON HAVE_AIO_H=1 HAVE_EXECINFO_H=1 HAVE_FCNTL_H=1 HAVE_GETLOGIN_R=1 HAVE_GETPWUID_R=1 HAVE_INTTYPES_H=1 HAVE_JOURNALD_H=TRUE HAVE_MATH_C99_LONG_DOUBLE=1 HAVE_POLL_H=1 HAVE_PTHREAD_MUTEX_TIMEDLOCK=ON HAVE_PTHREAD_MUTEX_TIMEDLOCK_LIBS= HAVE_PTHREAD_MUTEX_TIMEDLOCK_SYMBOL=1 HAVE_SYSLOG_H=1 HAVE_SYS_EVENTFD_H=1 HAVE_SYS_FILIO_H= HAVE_SYS_MODEM_H= HAVE_SYS_SELECT_H=1 HAVE_SYS_SOCKIO_H= HAVE_SYS_STRTIO_H= HAVE_SYS_TIMERFD_H=1 HAVE_TM_GMTOFF=1 HAVE_UNISTD_H=1 HAVE_XI_TOUCH_CLASS=1 WITH_ALSA=ON WITH_CAIRO=OFF WITH_CCACHE=ON WITH_CHANNELS=ON WITH_CLANG_FORMAT=ON WITH_CLIENT=ON WITH_CLIENT_AVAILABLE=1 WITH_CLIENT_CHANNELS=ON WITH_CLIENT_CHANNELS_AVAILABLE=1 WITH_CLIENT_COMMON=ON WITH_CLIENT_INTERFACE=OFF WITH_CUPS=ON WITH_DEBUG_ALL=OFF WITH_DEBUG_CAPABILITIES=OFF WITH_DEBUG_CERTIFICATE=OFF WITH_DEBUG_CHANNELS=OFF WITH_DEBUG_CLIPRDR=OFF WITH_DEBUG_DVC=OFF WITH_DEBUG_KBD=OFF WITH_DEBUG_LICENSE=OFF WITH_DEBUG_MUTEX=OFF WITH_DEBUG_NEGO=OFF WITH_DEBUG_NLA=OFF WITH_DEBUG_NTLM=OFF WITH_DEBUG_RAIL=OFF WITH_DEBUG_RDP=OFF WITH_DEBUG_RDPDR=OFF WITH_DEBUG_RDPEI=OFF WITH_DEBUG_RDPGFX=OFF WITH_DEBUG_REDIR=OFF WITH_DEBUG_RFX=OFF WITH_DEBUG_RINGBUFFER=OFF WITH_DEBUG_SCARD=OFF WITH_DEBUG_SND=OFF WITH_DEBUG_SVC=OFF WITH_DEBUG_SYMBOLS=OFF WITH_DEBUG_THREADS=OFF WITH_DEBUG_TIMEZONE=OFF WITH_DEBUG_TRANSPORT=OFF WITH_DEBUG_TSG=OFF WITH_DEBUG_TSMF=OFF WITH_DEBUG_TSMF=OFF WITH_DEBUG_TSMF_AVAILABLE=0 WITH_DEBUG_URBDRC=OFF WITH_DEBUG_WND=OFF WITH_DEBUG_X11=OFF WITH_DEBUG_X11_CLIPRDR=OFF WITH_DEBUG_X11_LOCAL_MOVESIZE=OFF WITH_DEBUG_XV=OFF WITH_DSP_EXPERIMENTAL=OFF WITH_DSP_FFMPEG=ON WITH_EVENTFD_READ_WRITE=1 WITH_FAAC=OFF WITH_FAAD2=OFF WITH_FFMPEG=TRUE WITH_FFMPEG=TRUE WITH_GFX_H264=ON WITH_GPROF=OFF WITH_GSM=OFF WITH_GSSAPI=OFF WITH_ICU=ON WITH_IPP=OFF WITH_JPEG=ON WITH_LAME=OFF WITH_LIBRARY_VERSIONING=ON WITH_LIBSYSTEMD=ON WITH_MACAUDIO=OFF WITH_MACAUDIO=OFF WITH_MACAUDIO_AVAILABLE=0 WITH_MANPAGES=ON WITH_MBEDTLS=OFF WITH_OPENCL=OFF WITH_OPENH264=OFF WITH_OPENSLES=OFF WITH_OPENSSL=ON WITH_OSS=ON WITH_PAM=ON WITH_PCSC=ON WITH_PROFILER=OFF WITH_PROXY=ON WITH_PROXY_MODULES=OFF WITH_PULSE=ON WITH_SAMPLE=OFF WITH_SANITIZE_ADDRESS=OFF WITH_SANITIZE_ADDRESS_AVAILABLE=1 WITH_SANITIZE_MEMORY=OFF WITH_SANITIZE_MEMORY_AVAILABLE=1 WITH_SANITIZE_THREAD=OFF WITH_SANITIZE_THREAD_AVAILABLE=1 WITH_SERVER=ON WITH_SERVER_CHANNELS=ON WITH_SERVER_INTERFACE=ON WITH_SHADOW=ON WITH_SMARTCARD_INSPECT=OFF WITH_SOXR=OFF WITH_SSE2=ON WITH_SWSCALE=ON WITH_THIRD_PARTY=OFF WITH_VAAPI=OFF WITH_VALGRIND_MEMCHECK=OFF WITH_VALGRIND_MEMCHECK_AVAILABLE=1 WITH_VERBOSE_WINPR_ASSERT=ON WITH_WAYLAND=ON WITH_WINPR_TOOLS=ON WITH_X11=ON WITH_XCURSOR=ON WITH_XDAMAGE=ON WITH_XEXT=ON WITH_XFIXES=ON WITH_XI=ON WITH_XINERAMA=ON WITH_XKBFILE=ON WITH_XRANDR=ON WITH_XRENDER=ON WITH_XSHM=ON WITH_XTEST=ON WITH_XV=ON WITH_ZLIB=ON
Build type:          None
CFLAGS:              -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -flto=auto -fPIC -Wall -Wno-unused-result -Wno-unused-but-set-variable -Wno-deprecated-declarations -fvisibility=hidden -Wimplicit-function-declaration -Wredundant-decls -g -fno-omit-frame-pointer -DWINPR_DLL
Compiler:            GNU, 12.1.0
Target architecture: x64

OS version connecting to (server side): Windows 10 Pro (Running in VMWare)
If available the log output from a run with /log-level:trace 2>&1 | tee log.txt
log.txt

Environment (please complete the following information):

OS: Garuda Linux, Arch based, KDE Plasma Wayland
Version/Distribution: N/A afaik (rolling release)
Architecture: [amd64, arm]: x64

@SoumyBhow do you have the chance to try this with current master?

@SoumyBhow do you have the chance to try this with current master?

I've replaced freerdp with freerdp-git from the AUR which from my understanding compiles and installs master but it didn't solve my issue.

In Cairo-Dock, RemoteApp Outlook starts with an icon, but than the icon gets replaces with a grey block and a question mark as it now detects a "rail:somenumber" applicaton. When I a set a custom icon, then it somehow magically returns the Outlook icon (not the custom icon). Maybe it has something to to with the switch to the "rail" part.


revert db init script change

It seemed pinning these dependencies wasn't necessary so removed them, but I just tried the benchmarks again and 23.04 versions of rmm and pylibraft were being pulled in causing the benchmarks to fail.
We still need 22.12 rapids version overall till we figure a10 issue on db so leaving that.

No need to build for these changes.

Do we need to change for Dataproc init scripts as well?

build


Copy endpoints support [WIP]

Hey, @alrocar this is a basic implementation of a COPY API client. I tried to make it match other interfaces as much as I could.
I know there's a little bit of duplication and lack of automated tests, but would like you to take a quick look and let me know how it looks overall.
(Most likely tests will fail, but it is not my fault (yet). Check the build history. I took a look and tried to figure out some of the errors but decided to move forward for the sake of completing the task at hand. I think most of the failures are due to changes in private API's).

PS: what I'm looking forward is to receiving some early feedback, before going any further, if possible. Thanks!

@rafatower thanks for taking this one! ❤
Looks good so far, adding here just some suggestions to simplify the usage:

Could not be cleaner to include the file handling inside the copyfrom and copyto methods? I'm thinking in something like this:

result = copyClient.copyfrom(query, 'copy_from.csv')

copyClient.copyto(query, 'export.csv')

Where the read and write happens inside the methods.

For the shake of simplicity I would suggest two changes in the way both APIs are used.

I don't want you to spend more time than necessary on this, so we can add this as a FR for next version:
We could make the query parameter optional for the copyfrom endpoint and derive it from the csv itself. So anyone looking to upload a CSV shouldn't learn about the COPY SQL syntax.
You could just write something like:
result = copyClient.copyfrom('copy_from.csv')
# this will read the header of the csv, build the COPY SQL, open the file and call the `copyfrom` endpoint.

We could also wrap the CopySqlClient.copyto API inside the Dataset resource.

So to export a table you would write something like this:
DATASET_ID = "world_borders"
dataset_manager = DatasetManager(auth_client)
dataset = dataset_manager.get(DATASET_ID)
dataset.copyto("world_borders.csv") # This would export the dataset to "world_borders.csv". We have a name attribute in the `dataset` so the file name could be optional as well and just write  `dataset.copyto()`

Two quick comments on your feedback:

I prefer to base the underlying API on streams/generators instead of
always assuming a file. Think of the cases of transferring data from a
foreign source or generating it
the query can be an arbitrary one. So instead of just exporting a table
you could for instance export a subset or a calculation.

Thanks for the prompt answer!
El vie., 27 jul. 2018 19:30, Alberto Romeu<EMAIL_ADDRESS>escribió:

@rafatower https://github.com/rafatower thanks for taking this one! ❤
Looks good so far, adding here just some suggestions to simplify the usage:

Could not be cleaner to include the file handling inside the copyfrom
and copyto methods? I'm thinking in something like this:

result = copyClient.copyfrom(query, 'copy_from.csv')
copyClient.copyto(query, 'export.csv')
Where the read and write happens inside the methods.

For the shake of simplicity I would suggest two changes in the way
both APIs are used.

I don't want you to spend more time than necessary on this, so we can add
this as a FR for next version:
We could make the query parameter optional for the copyfrom endpoint
and derive it from the csv itself. So anyone looking to upload a CSV
shouldn't learn about the COPY SQL syntax.
You could just write something like:
result = copyClient.copyfrom('copy_from.csv')
this will read the header of the csv, build the COPY SQL, open the file and call the copyfrom endpoint.

We could also wrap the CopySqlClient.copyto API inside the Dataset
resource.

So to export a table you would write something like this:
DATASET_ID = "world_borders"
dataset_manager = DatasetManager(auth_client)
dataset = dataset_manager.get(DATASET_ID)
dataset.copyto("world_borders.csv") # This would export the dataset to "world_borders.csv". We have a name attribute in the dataset so the file name could be optional as well and just write  dataset.copyto()
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/CartoDB/carto-python/pull/87#issuecomment-408486724,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB01pM0xfujNsSklsM-ZEZ-hj_RuiDjkks5uK04pgaJpZM4Vj34L
.

I tested manually with a script similar to the one in the /examples, but with a 1.2 GB input file.
execution logs
$ /usr/bin/time -v python copy_from_census_tracts.py
 04:26:23 PM - INFO - Dropping, re-creating and cartodbfy'ing the table through the Batch API...
 04:26:25 PM - INFO - Table created and cartodbfy'ed
 04:26:25 PM - INFO - COPY'ing the table FROM the file...
 04:29:25 PM - INFO - {u'total_rows': 73915, u'time': 179.017}
 04:29:25 PM - INFO - Table populated
        Command being timed: "python copy_from_census_tracts.py"
        User time (seconds): 2.47
        System time (seconds): 3.90
        Percent of CPU this job got: 3%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 3:01.34
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 19176
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 3382
        Voluntary context switches: 1698
        Involuntary context switches: 4128
        Swaps: 0
        File system inputs: 2360664
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

The point here is that the peak memory allocated by the python script was less than 20 MB (Maximum resident set size (kbytes): 19176).
So, it does not need to read the full CSV file in memory, but requests reads from the file object, and sends chunks through the network.

The way it works underneath (to me it was important to confirm this):

The file object is passed as data to the copyfrom method
The CopySQLClient uses the APIKeyAuthClient,  that in turn uses the BaseAuthClient from pyrestcli
The call to self.client.send is actually calling self.session.request which uses the requests library
That calls requests.Session.requests which calls to requests.Session.send which delegates the call to an adapter, particularly the HTTPAdapter
The data is transferred in chunks here

As you can see, the only requirement for the data is to be an interator of some sort. There's also a test for it here. The request.body is populated here BTW.

I did further tests and found out that it is not really streaming. It is sending the Content-Length header which disables the chunked transfer mode here.
This is a small piece of code but will need more time than I anticipated to get it right...

The Content-Length header is set here: https://github.com/requests/requests/blob/5aeec8b661f992050b1d034e727d7ce0326e9a56/requests/models.py#L475-L500
(I'm figuring half of this by patching and debugging with pdb)
I think this is closer to what we should do: https://gist.github.com/nbari/7335384
but it is a bit tough to integrate in a nice, "safe" and "usable" way. I keep digging...

I committed some sample code of how to send a chunked file through request and our library:
https://github.com/CartoDB/carto-python/pull/87/commits/3234f757bf01bb7c2cc217f7a966f2f742a7bf78#diff-99118b2871e39aceb0f505e083401249R103
It is meant to be eventually integrated in the sql module. But I'm having trouble. See the commit message: https://github.com/CartoDB/carto-python/pull/87/commits/3234f757bf01bb7c2cc217f7a966f2f742a7bf78 :cry:

With that particular file, I may have this issue:
$ file /home/rtorre/src/cartodb-tests/sql_copy_perf/tl_2014_census_tracts.csv
/home/rtorre/src/cartodb-tests/sql_copy_perf/tl_2014_census_tracts.csv: ASCII text, with very long lines, with CRLF, LF line terminators

ERROR:  unquoted carriage return found in data
HINT:  Use quoted CSV field to represent carriage return.
CONTEXT:  COPY tl_2014_census_tracts, line 2

a mixture of line terminators.

With a curl client I get the error clearly in headers and response body:
< HTTP/1.1 100 Continue
< HTTP/1.1 400 Bad Request
< Server: openresty
< Date: Thu, 02 Aug 2018 10:53:44 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 112
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: X-Requested-With, X-Prototype-Version, X-CSRF-Token
< vary: Authorization
< Content-Disposition: inline
< X-SQLAPI-Profiler: {"authorization":1,"getConnectionParams":1,"finish":1563,"total":1565}
< X-SQLAPI-Errors: {"hint":"Use quoted CSV field to represent carriage return ","statusCode":400,"message":"unquoted carriage return found in data"}
* HTTP error before end of send, stop sending
<
* Closing connection 0
{"error":["unquoted carriage return found in data"],"hint":"Use quoted CSV field to represent carriage return."}

shame on requests.

This trick from stackoverflow fixes the issue with mixed newline styles:
tr -d '\15\32' < windows.txt > unix.txt

With the file fixed, now it works (this is for a 1.2 GB file):
 01:00:19 PM - INFO - Dropping, re-creating and cartodbfy'ing the table through the Batch API...
 01:00:21 PM - INFO - Table created and cartodbfy'ed
 01:00:21 PM - INFO - COPY'ing the table FROM the file...
 01:03:51 PM - INFO - {u'total_rows': 73915, u'time': 209.43}
 01:03:51 PM - INFO - Table populated

:smile:

@javitonino and @juanignaciosl since you are the ones around I'm asking you for this review :pray:
Re. tests: most of the failing ones were broken before I touched this codebase :innocent: . But I created new E2E tests that do not work in travis because of AuthErrorException and the like. I'd appreciate any help in fixing those (I don't think I have permissions to check user, api keys and the rest of secrets).
They work in local with any api_key with enough privileges, both with python 2 and 3:
(env) [rtorre@fireflies carto-python (copy-endpoints-support)]$ python --version
Python 3.5.2
(env) [rtorre@fireflies carto-python (copy-endpoints-support)]$ py.test  tests/test_sql_copy.py
================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.5.2, pytest-3.7.1, py-1.5.4, pluggy-0.7.1
rootdir: /home/rtorre_unencrypted/src/carto-python, inifile:
plugins: requests-mock-1.5.2
collected 8 items

tests/test_sql_copy.py ........                                                                                [100%]

================================================================================================================================ 8 passed in 5.30 seconds ================================================================================================================================

(env2) [rtorre@fireflies carto-python (copy-endpoints-support)]$ python --version
Python 2.7.12
(env2) [rtorre@fireflies carto-python (copy-endpoints-support)]$ py.test tests/test_sql_copy.py
================================================================================================================================== test session starts ===================================================================================================================================
platform linux2 -- Python 2.7.12, pytest-3.7.1, py-1.5.4, pluggy-0.7.1
rootdir: /home/rtorre_unencrypted/src/carto-python, inifile:
plugins: requests-mock-1.5.2
collected 8 items

tests/test_sql_copy.py ........                                                                                [100%]

================================================================================================================================ 8 passed in 5.06 seconds ================================================================================================================================

About the other tests, I'm not really in a position of fixing them. Hopefully they fail because of the same perms issue.
My original intent was to release this branch this week.
Thanks in advance for the review and the help with the tests...

I got a look at the nginx logs and the API key being used was deleted or re-generated. I'll do my best to fix it, but it is a bit of a PITA without access to the travis secrets. Any help is appreciated.

I'll take a look at this later, but I don't have access to the build settings at Travis. Maybe @alberhander can help with that.

I got access and changed the master key in the config. In general tests do table creation/deletion and otherwise they get a 401 error. Let's see the results in the next round...

PS: BTW, nice work with the tests! :muscle:

Wonderful examples 👍

I think I covered all the comments (thanks!) and now this is ready for release.
@juanignaciosl I'd do it myself but I can't yet, if you could help me... :hugs:

@rafatower ok, let me know when you're around and we'll pair this.


integration_tests: update tidb version

Signed-off-by: HuSharp<EMAIL_ADDRESS>In order to merge https://github.com/pingcap/tidb/pull/40955, we introduce the https://github.com/tikv/client-go/pull/683 to do some temporary changes. This PR cleans up these temporary changes.

/retest

updated to latest master in https://github.com/tikv/client-go/pull/695 , closing this one.


add WETH option design for maker

First commit on add maker WETH option.

Also add test for takerToken == WETH but maker receive ETH?

Also add test for takerToken == WETH but maker receive ETH?

I thought this case is already tested in testFillRFQWithWETH ? 🤔