check_limits does not work well with pyHKL object

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/simulators.py
checking to see if check_limits was installed. it is, but it balks at tardis.l not having limits, when the tardis.gamma is the one that prevented the scan from going.  Most  people do not want limits on H, K , L but on the diffractometer.
In [201]: RE(scan(dets,tardis.l,-0.025,0.55,24, md={'purpose':'test L'}))
Transient Scan ID: 111588     Time: 2018/11/18 02:50:57
Persistent Unique Scan ID: '87bf87b9-fce7-41e3-bf39-8211ddfb9203'
/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/device.py:738: UserWarning: 'signal_names' has been renamed 'component_names' for clarity because it may include a mixture of Signals and Devices -- any Components. This alias may be removed in a future release of ophyd.
  warnings.warn("'signal_names' has been renamed 'component_names' for "
New stream: 'baseline'
---------------------------------------------------------------------------
LimitError                                Traceback (most recent call last)
~/Beamline/ScienceComm/2018_11_MuR/scripts.py in <module>()
----> 1 RE(scan(dets,tardis.l,-0.025,0.55,24, md={'purpose':'test L'}))

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    668                     # it (unless it is a canceled error)
    669                     if exc is not None:
--> 670                         raise exc
    671
    672             if self._interrupted:

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in _run(self)
   1118             self.log.error("Run aborted")
   1119             self.log.error("%r", err)
-> 1120             raise err
   1121         finally:
   1122             # Some done_callbacks may still be alive in other threads.

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in _run(self)
   1017                         resp = self._response_stack.pop()
   1018                         try:
-> 1019                             msg = self._plan_stack[-1].send(resp)
   1020                         # We have exhausted the top generator
   1021                         except StopIteration:

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1236         plan = monitor_during_wrapper(plan, self.monitors)
   1237         plan = baseline_wrapper(plan, self.baseline)
-> 1238         return (yield from plan)

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1092         return (yield from plan)
   1093     else:
-> 1094         return (yield from plan_mutator(plan, insert_baseline))
   1095
   1096

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    136                     continue
    137                 else:
--> 138                     raise ex
    139         # if inserting / mutating, put new generator on the stack
    140         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     89             ret = result_stack.pop()
     90             try:
---> 91                 msg = plan_stack[-1].send(ret)
     92             except StopIteration as e:
     93                 # discard the exhausted generator

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    722     plan1 = plan_mutator(plan, insert_after_open)
    723     plan2 = plan_mutator(plan1, insert_before_close)
--> 724     return (yield from plan2)
    725
    726

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    136                     continue
    137                 else:
--> 138                     raise ex
    139         # if inserting / mutating, put new generator on the stack
    140         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     89             ret = result_stack.pop()
     90             try:
---> 91                 msg = plan_stack[-1].send(ret)
     92             except StopIteration as e:
     93                 # discard the exhausted generator

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    136                     continue
    137                 else:
--> 138                     raise ex
    139         # if inserting / mutating, put new generator on the stack
    140         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     89             ret = result_stack.pop()
     90             try:
---> 91                 msg = plan_stack[-1].send(ret)
     92             except StopIteration as e:
     93                 # discard the exhausted generator

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    780     plan1 = plan_mutator(plan, insert_after_open)
    781     plan2 = plan_mutator(plan1, insert_before_close)
--> 782     return (yield from plan2)
    783
    784

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    136                     continue
    137                 else:
--> 138                     raise ex
    139         # if inserting / mutating, put new generator on the stack
    140         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     89             ret = result_stack.pop()
     90             try:
---> 91                 msg = plan_stack[-1].send(ret)
     92             except StopIteration as e:
     93                 # discard the exhausted generator

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    136                     continue
    137                 else:
--> 138                     raise ex
    139         # if inserting / mutating, put new generator on the stack
    140         # and replace the current msg with the first element from the

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     89             ret = result_stack.pop()
     90             try:
---> 91                 msg = plan_stack[-1].send(ret)
     92             except StopIteration as e:
     93                 # discard the exhausted generator

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/plans.py in scan(detectors, motor, start, stop, num, per_step, md)
    261             yield from per_step(detectors, motor, step)
    262
--> 263     return (yield from inner_scan())
    264
    265

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
    962                 plan = gen_func(*inner_args, **inner_kwargs)
    963                 plan = wrapper(plan, *args, **kwargs)
--> 964                 return (yield from plan)
    965             return dec_inner
    966         return dec

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    871         return (yield from plan)
    872
--> 873     return (yield from finalize_wrapper(inner(), unstage_devices()))
    874
    875

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    432     cleanup = True
    433     try:
--> 434         ret = yield from plan
    435     except GeneratorExit:
    436         cleanup = False

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in inner()
    869     def inner():
    870         yield from stage_devices()
--> 871         return (yield from plan)
    872
    873     return (yield from finalize_wrapper(inner(), unstage_devices()))

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
    962                 plan = gen_func(*inner_args, **inner_kwargs)
    963                 plan = wrapper(plan, *args, **kwargs)
--> 964                 return (yield from plan)
    965             return dec_inner
    966         return dec

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in run_wrapper(plan, md)
    294     yield from contingency_wrapper(plan,
    295                                    except_plan=except_plan,
--> 296                                    else_plan=close_run)
    297     return rs_uid
    298

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in contingency_wrapper(plan, except_plan, else_plan, final_plan, pause_for_debug)
    494     cleanup = True
    495     try:
--> 496         ret = yield from plan
    497     except GeneratorExit:
    498         cleanup = False

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/plans.py in inner_scan()
    259     def inner_scan():
    260         for step in steps:
--> 261             yield from per_step(detectors, motor, step)
    262
    263     return (yield from inner_scan())

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/plan_stubs.py in one_1d_step(detectors, motor, step)
    858         yield Msg('wait', None, group=grp)
    859
--> 860     yield from move()
    861     return (yield from trigger_and_read(list(detectors) + [motor]))
    862

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/plan_stubs.py in move()
    855         grp = _short_uid('set')
    856         yield Msg('checkpoint')
--> 857         yield Msg('set', motor, step, group=grp)
    858         yield Msg('wait', None, group=grp)
    859

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    161         try:
    162             # yield out the 'current message' and collect the return
--> 163             inner_ret = yield msg
    164         except GeneratorExit:
    165             # special case GeneratorExit.  We must clean up all of our plans

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in _run(self)
   1068                         # exceptions (coming in via throw) can be
   1069                         # raised
-> 1070                         response = yield from coro(msg)
   1071                     # special case `CancelledError` and let the outer
   1072                     # exception block deal with it.

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/asyncio/coroutines.py in coro(*args, **kw)
    208         @functools.wraps(func)
    209         def coro(*args, **kw):
--> 210             res = func(*args, **kw)
    211             if (base_futures.isfuture(res) or inspect.isgenerator(res) or
    212                 isinstance(res, CoroWrapper)):

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in _set(self, msg)
   1822         group = kwargs.pop('group', None)
   1823         self._movable_objs_touched.add(msg.obj)
-> 1824         ret = msg.obj.set(*msg.args, **kwargs)
   1825         p_event = asyncio.Event(loop=self.loop)
   1826         pardon_failures = self._pardon_failures

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/positioner.py in set(self, new_position, timeout, moved_cb, wait)
     83         """
     84         return self.move(new_position, wait=wait, moved_cb=moved_cb,
---> 85                          timeout=timeout)
     86
     87     def stop(self, *, success: bool =False):

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/pseudopos.py in move(self, pos, **kwargs)
    173         '''
    174         self._target = pos
--> 175         return self._parent.move_single(self, pos, **kwargs)
    176
    177     def describe(self):

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/pseudopos.py in move_single(self, pseudo, position, **kwargs)
    748         target = list(self.target)
    749         target[idx] = position
--> 750         return self.move(self.PseudoPosition(*target), **kwargs)
    751
    752     @property

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/pseudopos.py in wrapped(self, *args, **kwargs)
    196             pos, new_kwargs = m(*args, **kwargs)
    197
--> 198             return method(self, pos, **new_kwargs)
    199
    200         return wrapped

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/pseudopos.py in move(self, position, wait, timeout, moved_cb)
    850                 positioner._target = single_pos
    851         return super().move(position, wait=wait, timeout=timeout,
--> 852                             moved_cb=moved_cb)
    853
    854     move.__doc__ = SoftPositioner.move.__doc__

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/positioner.py in move(self, position, wait, timeout, moved_cb)
    332             If motion fails other than timing out
    333         '''
--> 334         status = super().move(position, moved_cb=moved_cb, timeout=timeout)
    335
    336         self._setup_move(position, status)

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/positioner.py in move(self, position, moved_cb, timeout)
    178             timeout = self._timeout
    179
--> 180         self.check_value(position)
    181
    182         self._run_subs(sub_type=self._SUB_REQ_DONE, success=False)

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/pseudopos.py in check_value(self, pseudo_pos)
    616         real_pos = self.forward(pseudo_pos)
    617         for real, pos in zip(self._real, real_pos):
--> 618             real.check_value(pos)
    619
    620     @property

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/epics_motor.py in check_value(self, pos)
    217     def check_value(self, pos):
    218         '''Check that the position is within the soft limits'''
--> 219         self.user_setpoint.check_value(pos)
    220
    221     def _pos_changed(self, timestamp=None, value=None, **kwargs):

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/ophyd/signal.py in check_value(self, value)
    829         if not (low_limit <= value <= high_limit):
    830             raise LimitError('Value {} outside of range: [{}, {}]'
--> 831                              .format(value, low_limit, high_limit))
    832
    833     @raise_if_disconnected

LimitError: Value -3.539555034426563 outside of range: [-2.9, 178.0]

In [202]: check_limits(scan(dets,tardis.l,-0.025,0.55,24, md={'purpose':'test L'}))
/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/simulators.py:109: UserWarning: Limits are not set on tardis_l
  warn("Limits are not set on {}".format(msg.obj.name))

@ambarb Is this a scan that should have failed due to out-of-limits or failed in error?

For this particular scan, I believe that once it did fail because tardis.gamma was outside the range allowed.  I could not check because check_limits did not like that tardis.l had no limits.
So in the end, my work flow was:
try check_limits
see that this didn't work
put the scan into the RE.
it look like it started, because the baseline process takes so long
I looked away and concentrated on something else and found later that the scan never started.  Check_limits that does not require everything to have limits or some eqivalent of "on_sim" in spec is needed to stream line this process

"Transferred" to https://github.com/NSLS-II/hklpy/issues/13.


Bad error message for package-qualified imports with no registered shorthand

If I do e.g. imports [foo.Blah] but there is no foo shorthand defined in my root module's packages section, I should get a nice error. Currently I get a panic!

Currently there isn't even a panic, the executing program hangs indefinitely it seems.

No longer panics or hangs.
Current output for reference:
The package shorthand 'foo' that you are using in the 'imports' section of the header of module 'main.roc' doesn't exist.
Check that package shorthand is correct or reference the package in an 'app' or 'package' header.

Probably can be closed.


Bump version to 2.0.0

This release contains breaking changes.
To convert your old prototypes to use the new kit, follow this guide.
2.0.0

Update govuk elements sass (#124)
Update the prototype kit to use Nunjucks for templating (#123)
Create config file that stores prototype configuration (#120)
Add phase banner includes (#118)
Use npm start as the standard way to run the app (#111)
Add warning if folder missing or module missing (#100)
Improve error handling around port in use, find new port (#95)
Add body-parser for parsing POSTs (#86)
Add question page (#72)
Add js for toggled content (#70)
Add Start Page (#45)
Add Check Your Answers page (#36)
Add confirmation page (#35)
Upgraded to Express 4 (#32)
Add jQuery to the kit, so it's available on all pages by default (#18)
Add page without header and footer (#12)

Will you also need to add a tag for this version?

I think the changelog could be clearer in highlighting significant changes. See (Nunjuks')[https://github.com/mozilla/nunjucks/blob/master/CHANGELOG.md] changelog.
In particular it would be good to mention what the breaking changes are, and a bit more about Nunjucks. If we have a migration guide, it would be good to link to that.

Tagging is automatic once we've merged. Don't merge just yet though...

Merged #126


ElementReference causes "Invalid JSON" exception when returned to JavaScript

Describe the bug
Calling C# from JavaScript, then JavaScript from C# to obtain ElementReferences, then returning those ElementReferences to C#, Blazor throws an exception:
blazor.webassembly.js:1 Uncaught (in promise) Error: System.Text.Json.JsonException: Invalid JSON
  at Microsoft.JSInterop.Infrastructure.DotNetDispatcher.EndInvokeJS (/Microsoft.JSInterop.JSRuntime jsRuntime, System.String arguments) <0x20ed888 + 0x000fc> in <4e424253df194e1299a1044d93a79164>:0
  at Mono.WebAssembly.Interop.MonoWebAssemblyJSRuntime.EndInvokeJS (/System.String argsJson) <0x20e4a58 + 0x0000a> in <d0b620549c284928acbf6b89d371ca02>:0
    at Object.callMethod (blazor.webassembly.js:1)
    at Object.endInvokeJSFromDotNet (blazor.webassembly.js:1)
    at blazor.webassembly.js:1

To Reproduce
Steps to reproduce the behavior:

.Net Core 3.0.0-100
dotnet new blazorwasm

Create a JavaScript method that creates an element:
window.CreateElement = (text) => {
  var element = document.createElement("span");
  element.innerText = text;
  return element;
}

Create a [JSInvokable] C# method:
[JSInvokable]
public static async Task<List<ElementReference>> GetElements(){
  List<ElementReference> refs = new List<ElementReference>();
  refs.Add(await JSInterop.InvokeAsync<ElementReference>("CreateElement","first"));
  refs.Add(await JSInterop.InvokeAsync<ElementReference>("CreateElement","second"));
  return refs;
}

Create another JavaScript function that calls the C# method:
window.Test = async () => {
  var refs = await DotNet.InvokeAsync("TestProject","GetElements");
}

Invoke window.Test.
Expected behavior
Expect no exception to be thrown, and for the elements created by the CreateElement method to be returned to the Test method.

Thanks for contacting us, @HughPH.
ElementReference is a reference to a DOM element that Blazor has rendered. It cannot be used to refer an element rendered by JavaScript as Blazor has no knowledge of it. You can pass the element id from JavaScript to Blazor and then use it as string internally.

@mkArtakMSFT Thanks for the reply, that makes sense. I had expected that Blazor would 'foster'  (temporarily adopt) unknown elements and create a new _internalId for them, keeping them immutable but referable, then 'translate' them back to their JavaScript object references when being passed back across the divide.


Implement htsget id resolver interface

When a query is sent to htsget via, for instance, GET /reads/<id>, the spec says that id can be any of those things:

The spec is intentionally loose in this regard since different organizations might have differing requirements regarding how this endpoints and internal (organizational) structure they have in place.
A "resolver" interface should allow for those organizations to easily "plug and play" htsget into their systems more easily and/or filter what's a legal ID or not, perhaps via regexps like the reference implementation does.

Not necessarily DRS reference implementation, but could be useful to examine: https://github.com/ncbi/ncbi-drs.
@victorskl @AndrewPattison Keep this issue in the back of your mind(s), we might need to discuss to which extent we implement this and which assumptions we do on the htsget-rs side?

For now this will entail implementing support for regexp-based mapping of resources:
https://github.com/ga4gh/htsget-refserver/blob/main/internal/htsconfig/datasources.go
BUT, the resolver interface (perhaps another crate named htsget-resolver) should be modular and accomodate other disambiguation mechanisms when trying to access a dataset. On top of the regexp mechanism outlined above, this interface could be based on URL schemes:
s3://...  (AWS)
gs://...  (Google)
gds://...(Illumina's ICA)
drs://... (GA4GH)
file://... (Local)
https://... (On-prem server or AWS presigned URLs)

Assigning this one to you, @CastilloDel, I think that @mmalenic will be busy with the S3 storage backend modeling since I've introduced a couple of extra requirements there.

I'm thinking the easiest  approach would be to configure it in a way similar to how the reference implementation does it, but using environment variables. Then the htsget-http-core could provide helpers to build an id resolver struct (from a htsget-id-resolver crate) and the LocalStorage would store the id resolver and use it when it is needed.

https://crates.io/crates/regex seems like the way to go

Yes! Sounds good, other ID resolvers are getting stabilised still in specs,
so this is a good start indeed, thanks Daniel!
On Fri, 13 Aug 2021 at 23:05, Daniel del Castillo @.***>
wrote:

https://crates.io/crates/regex seems like the way to go
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/umccr/htsget-rs/issues/10#issuecomment-898443868, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AABK3Y4KDFJYFOFN6W2TQNTT4UKBNANCNFSM42U2AHLQ
.


[Question] Validation when doing partial updates?

I noticed that the update mutations expect the full object passed in. Is this because of the optimistic updates? How should we best handle cases where we only want to update part of the schema? Use .partial()?
These are the relevant bits of code I am referring to
export const updateReportSchema = baseSchema;
export const updateReportParams = baseSchema
  .extend({
    patientId: z.coerce.string().min(1),
  })
  .omit({
    userId: true,
  });

export const updateReport = async (
  id: ReportId,
  report: UpdateReportParams,
) => {

Interested in this also


New release?

There are some long-unreleased changes. Looks like we can release them by simply pushing a tag.
I have the permission to do this, but I want someone's approval.

what does a release tag mean in practise for webxdc-dev?
if it is not much more than a formal thing, i'd say, go for it :) there are lots of improvements that are worth rewarded by a release (i personally do not used webxdc-dev currently, so i cannot judge the overall state)

what does a release tag mean in practise for webxdc-dev?

When you push a tag, it publishes it to https://www.npmjs.com/

i see. recent changes in README and updated dependencies - and older changes are tested meanwhile by you - so, i'd say go for a new release :)

I forgot to mention the changelog and package.json. After #39 it's really a matter of pushing a tag.


Update helm

Tool name
helm
Tool license
Apache 2.0
Add or update?

[ ] Add
[X] Update

Desired version
3.6.2
Approximate size
No response
Brief description of tool
There is a bug in helm v3.6.1 which breaks compatibility with azure-cli (az acr helm repo add) and forces users to use --pass-credentials when adding an Azure Container Registry. Helm v3.6.2 fixes this issue.
URL for tool's homepage
No response
Provide a basic test case to validate the tool's functionality.
No response
Virtual environments affected

[ ] Ubuntu 16.04
[ ] Ubuntu 18.04
[ ] Ubuntu 20.04
[ ] macOS 10.15
[ ] macOS 11
[ ] Windows Server 2016
[ ] Windows Server 2019

Can this tool be installed during the build?
No response
Tool installation time in runtime
No response
Are you willing to submit a PR?
No response

Hello @josefschabasser. Thank you for your report. We update images on weekly basis. Updated Help should be available on all images in about 2 weeks (next week is holidays week). I'll post an update message. Thank you.

Hello @josefschabasser. Helm was successfully updated to 3.6.2 version across all the images. I'm going to close this issue. In case of any questions feel free to contact us.


Minor typo fixes

Fixed a few typos.

@GitHubPang Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

Thank you!


I deploy a Keras model,but get a pickle.PicklingError

I want to deploy a Keras model,code as below:
from clipper_admin import ClipperConnection,DockerContainerManager

clipper_conn = ClipperConnection(DockerContainerManager())
try:
	clipper_conn.start_clipper()
except Exception, e:
	print(e)
	clipper_conn.stop_all()

clipper_conn.register_application(name="hello-keras", input_type="bytes", default_output="this is default output", slo_micros=100000)

#clipper_conn.get_all_apps()
from keras.models import load_model
model = load_model('./models/model.h5')

def predict(inputs):
    import base64
    import io
    import tempfile
    import numpy as np
    from keras.preprocessing.image import load_img, img_to_array

    num_imgs = len(inputs)
    preds = []
    for i in range(num_imgs):
        # Create a temp file to write to
        tmp = tempfile.NamedTemporaryFile('wb', delete=False, suffix='.png')
        tmp.write(io.BytesIO(inputs[i]).getvalue())
        tmp.close()

	#read image and convert image
        image = load_img(tmp.name,grayscale=True,target_size=(28,28))
	image = img_to_array(image)
	image = np.expand_dims(image, axis=0)
	image = image.astype('float32')
	image /= 255

        #predict
	pred = model.predict(image)
        preds.append(pred)
    return [str(p) for p in preds]

from clipper_admin.deployers.python import deploy_python_closure

deploy_python_closure(clipper_conn, name="keras-model", version=1, input_type="bytes", func=predict,base_image='docker.io/clipper/keras-container:develop',registry="docker.io/clipper",pkgs_to_install=[])

clipper_conn.link_model_to_app(app_name="hello-keras", model_name="keras-model")

But when I run this script,I get a error.
Traceback (most recent call last):
  File "start_keras.py", line 46, in <module>
    deploy_python_closure(clipper_conn, name="keras-model", version=1, input_type="bytes", func=predict,base_image='docker.io/clipper/keras-container:develop',registry="docker.io/clipper",pkgs_to_install=[])
  File "/usr/lib/python2.7/site-packages/clipper_admin/deployers/python.py", line 189, in deploy_python_closure
    serialization_dir = save_python_function(name, func)
  File "/usr/lib/python2.7/site-packages/clipper_admin/deployers/deployer_utils.py", line 37, in save_python_function
    c.dump(func)
  File "/usr/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 272, in dump
    raise pickle.PicklingError(msg)
pickle.PicklingError: Could not pickle object as excessively deep recursion required.

What's wrong with my predict function?Couldn't I deploy a Keras model in this way?
Thanks in advance.If you have a example of deploying Keras model.I will be very grateful.

Hi @dyoung23,
I noticed you put the base_image as base_image='docker.io/clipper/keras-container:develop. As I can recall, we never wrote a custom keras container. Did you write it? If so, how?
This error message probably comes from this line in your predict function:
pred = model.predict(image)

Precisely, what happened was:

cloudpickle tries to serialize model object
model object is referring to the global model object, which is a compiled model instance.
Keras is a pure python library; however, when the object was compiled, it instantiates tensorflow or theano object in the background.
Keras can be pickled, but tensorflow can't.
This is why serialization failed.

Steps to resolve this:

There need to be a custom keras container just like our mxnet, pytorch, and tensorflow container.
The code most boilderplate, the idea here is that you should instantiate and compile the model in the beginning of the model container (when the model was actually ran)
The deployer code can be found [here](https://github.com/ucbrise/clipper/blob/develop/clipper_admin/clipper_admin/deployers]. Using tensorflow deployer as an example,  the interface becomes:

        # Note that predict function now takes sess as an argument
        # In Keras, this will be "model"
        def predict(sess, inputs):
            # In Keras, this will be model.predict
            preds = sess.run('predict_class:0', feed_dict={'pixels:0': inputs})
            return [str(p) for p in preds]

       deploy_tensorflow_model(
            clipper_conn,
            model_name,
            version,
            input_type,
            predict_fn,
            sess)

@simon-mo Thank you for your answer.
I saw a example which deploy a pure python model in sklearn. So I follow it  and deploy a Keras model in same way,but fail to realize the effort of Tensorflow.
I build this container in this Dockerfile and rewrite keras_container.py.
I think it's wrong way,after your answer.
In fact,before I try this way,I thought using the method as your recommend,but tensorflow deployer will takes sess or path to model files as an argument, and copy your model file into the container will be started.But this function will check if all the model files are present,so it can't be used to deploy keras model directly.Should I write a keras deployer like this deployer or other methods.
Thank you.

@dyoung23 We've received a lot of interest in a Keras deployer, so we were wondering if you would be interested in contributing to Clipper by submitting a PR for your deployer.

@simon-mo
I have recently implemented a Keras deployer by rewriting the Tensorflow one
just opened a PR for this #594

@dyoung23 https://github.com/ucbrise/clipper/pull/594 has been merged thanks to @eavidan's efforts!


Job preemption across cluster queue doesn't work for apiVersion v1beta1

What happened:
Following the tutorial, I have set up the WorkloadPriorityClass with high and low priority (tutorial) and set up two ClusterQueues with preemption enabled (tutorial) within the same cohort but I can't make the high priority job in cluster queue a to preempt the low priority job in cluster queue b.
What you expected to happen:
I am expecting the high priority job in cluster queue a will preempt the low priority job in cluster queue b.
How to reproduce it (as minimally and precisely as possible):
Command to run
# define queues
kubectl apply -f cluster-queue-cohort-a.yaml -f cluster-queue-cohort-b.yaml
kubectl apply -f default-flavor.yaml
kubectl create namespace cohort-3
kubectl apply -f default-user-queue-cohort-a.yaml -f default-user-queue-cohort-b.yaml
# define priority class
kubectl apply -f high-priority.yaml -f low-priority.yaml

# run low priority job which requires 4 cpu in cluster queue b with cpu quota of 4
kubectl create -f sample-job-low-priority-cohort-b.yaml

# run high priority job which requires 6 cpu in cluster queue a with cpu quota of 4
kubectl create -f sample-job-high-priority-cohort-a.yaml

yaml file definition
high-priority.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
  name: high-priority
value: 10000

low-priority.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
  name: low-priority
value: 100

cluster-queue-cohort-a.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: "cluster-queue-cohort-a-3"
spec:
  namespaceSelector: {} # match all.
  cohort: "team-ab-3"
  preemption:
    reclaimWithinCohort: Any
    borrowWithinCohort:
      policy: LowerPriority
      maxPriorityThreshold: 5000
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 4
      - name: "memory"
        nominalQuota: 36Gi

cluster-queue-cohort-b.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: "cluster-queue-cohort-b-3"
spec:
  namespaceSelector: {} # match all.
  cohort: "team-ab-3"
  preemption:
    reclaimWithinCohort: Any
    borrowWithinCohort:
      policy: LowerPriority
      maxPriorityThreshold: 5000
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 4
      - name: "memory"
        nominalQuota: 36Gi

default-flavor.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: "default-flavor"

default-user-queue-cohort-a.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  namespace: "cohort-3"
  name: "user-queue-cohort-a-3"
spec:
  clusterQueue: "cluster-queue-cohort-a-3"

default-user-queue-cohort-b.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  namespace: "cohort-3"
  name: "user-queue-cohort-b-3"
spec:
  clusterQueue: "cluster-queue-cohort-b-3"

sample-job-high-priority-cohort-a.yaml
apiVersion: batch/v1
kind: Job
metadata:
  generateName: sample-job-
  namespace: cohort-3
  labels:
    kueue.x-k8s.io/queue-name: user-queue-cohort-a-3
    kueue.x-k8s.io/priority-class: high-priority
spec:
  parallelism: 6
  completions: 6
  suspend: true
  template:
    spec:
      containers:
      - name: dummy-job
        image: gcr.io/k8s-staging-perf-tests/sleep:v0.1.0
        args: ["15s"]
        resources:
          requests:
            cpu: 1
            memory: "200Mi"
      restartPolicy: Never

sample-job-low-priority-cohort-b.yaml
apiVersion: batch/v1
kind: Job
metadata:
  generateName: sample-job-
  namespace: cohort-3
  labels:
    kueue.x-k8s.io/queue-name: user-queue-cohort-b-3
    kueue.x-k8s.io/priority-class: low-priority
spec:
  parallelism: 4
  completions: 4
  suspend: true
  template:
    spec:
      containers:
      - name: dummy-job
        image: gcr.io/k8s-staging-perf-tests/sleep:v0.1.0
        args: ["15s"]
        resources:
          requests:
            cpu: 1
            memory: "200Mi"
      restartPolicy: Never

Anything else we need to know?:
Environment:

Kubernetes version (use kubectl version):

Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.2-eks-db838b0

Kueue version (use git describe --tags --dirty --always): 0.8.0
Cloud provider or hardware configuration: Amazon EKS
OS (e.g: cat /etc/os-release):
Kernel (e.g. uname -a):
Install tools: see https://kueue.sigs.k8s.io/docs/installation/#install-a-released-version
Others:

@trasc could you help to check if this is indeed a bug ? Thank you!

In short, the low priority job is consider for eviction because the cluster queue in which it runs is not borrowing.
Some additional detail on preempt-within-cohort-while-borrowing can be found in the features KEP.
To get to the scenario I think you are targeting, you can add a third cluster-queue that adds 1 cpu  to the cohort
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: "cluster-queue-cohort-c-3"
spec:
  namespaceSelector: {} # match all.
  cohort: "team-ab-3"
  preemption:
    reclaimWithinCohort: Any
    borrowWithinCohort:
      policy: LowerPriority
      maxPriorityThreshold: 5000
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["cpu", "memory"]
    flavors:
    - name: "default-flavor"
      resources:
      - name: "cpu"
        nominalQuota: 1
      - name: "memory"
        nominalQuota: 36Gi

and increase the resource requirements of the low priority job
apiVersion: batch/v1
kind: Job
metadata:
  generateName: sample-job-
  namespace: cohort-3
  labels:
    kueue.x-k8s.io/queue-name: user-queue-cohort-b-3
    kueue.x-k8s.io/priority-class: low-priority
spec:
  parallelism: 5
  completions: 5
  suspend: true
  template:
    spec:
      containers:
      - name: dummy-job
        image: gcr.io/k8s-staging-perf-tests/sleep:v0.1.0
        args: ["15s"]
        resources:
          requests:
            cpu: 1
            memory: "200Mi"
      restartPolicy: Never

Please let me know which section of the documentation we can improve to make this behavior a bit more clear.

/remove-kind bug

/kind documentation

@trasc Thank you for your instruction. I tried your example and it works.
I think it would be good to mention clearly about the preemption in Kueue is guaranteed quota vs over-quota. Only the job that is served with over-quota will be preempted by other jobs in the same cohort. The only place mentioned about the rule is https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/#preemption

only preempt Workloads in the cohort that have lower priority than the pending Workload.
which doesn't clearly express the actual behavior.


Fix issues

@hmgaudecker  there seems to be an issue in the test of the pre-commits that only concerns linux, and also why are you changing our pre-commit file all the time while I am assigned to that ...!!!!

Not sure where to put the stuff on git-posh ... does not seem to fit in the templates as they are not there to help windows user to use PowerShell...

Not sure where to put the stuff on git-posh ... does not seem to fit in the templates as they are not there to help windows user to use PowerShell...

Well, they are there to help novices use their system efficiently. Due to popular demand, I would thus consider it in scope.
Just call the section "Windows installation notes and tricks." ?

@hmgaudecker there seems to be an issue in the test of the pre-commits that only concerns linux,

Okay, this seemed to be caused by c.communicate() returning bytes objects rather than strings and the code not being well prepared for that.
Please double-check what I did in bc86830. Fixes it on my local machine.
May or may not be related: On my local machine, the waf + R tests fail because the environment does not seem to get activated. So python waf.py configure does not find R

and also why are you changing our pre-commit file all the time while I am assigned to that ...!!!!

All the time? :-)  You mean once until looking at this in 26bb799? :-) Just tried to be nice while I did the same over at gettsim

The error that Azure throws for bc8683063aa on both MacOS and Linux seems to be specific to their setup, some temporary file that gets re-used.
E        +  where None = <function search at 0x106d59b00>('(Passed|Failed)', 'b"An unexpected error has occurred: FileExistsError: [Errno 17] File exists: \'/Users/vsts/.cache/pre-commit\'\\nCheck the log at /Users/vsts/.cache/pre-commit/pre-commit.log\\n"')

Too late to investigate that today, can you please check at some point? Thanks!

The issue with R is, of course, caused by me having a too recent conda on my system, so that source activate env does not work.
I just googled azure "conda init" and came up with this setup -- can we shamelessly borrow from there? That should help on the Windows testing, too.

The issue with R is, of course, caused by me having a too recent conda on my system, so that source activate env does not work.
I just googled azure "conda init" and came up with this setup -- can we shamelessly borrow from there? That should help on the Windows testing, too.

I somehow gave up hope of full testing on windows, because also it is very difficult to install a Latex distribution via the command line,. But we could in general run, the pre-commit tests and the cookiecutter tests.

So I thinking you reimplemented some flake8 pre commit stuff; that we actually excluded on purpose. See error trailing comma missing... can I just revert your changes?  I just used the other setup for consistency, if we had good reasons to deviate, we should continue to do so.

@raholler: Can we release 0.3 today?

I am in an ICE from 3 onwards, then I will check everything again, then we can release, if thats ok for you

So long as the updated installation instructions are live by 9am tomorrow morning, I shall be more than happy. Safe travels!


Added Spanish translation

I think there is no need in double quotes in this phrase - "Descripción "

Fixed.


[SPARK-23941][Mesos] Mesos task failed on specific spark app name

What changes were proposed in this pull request?
Shell escaped the name passed to spark-submit and change how conf attributes are shell escaped.
How was this patch tested?
This test has been tested manually with Hive-on-spark with mesos or with the use case described in the issue with the sparkPi application with a custom name which contains illegal shell characters.
With this PR, hive-on-spark on mesos works like a charm with hive 3.0.0-SNAPSHOT.
I state that this contribution is my original work and that I license the work to the project under the project’s open source license

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Test build #89077 has started for PR 21014 at commit fb078eb.

Test build #89077 has finished for PR 21014 at commit fb078eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89077/
Test PASSed.

IMO, mainClass doesn't need the same because space is not allowed but pyFiles may have the same fix because it's a path. Do you want me to apply the shellEscape as well for pyFiles?

mainClass doesn't need the same because space is not allowed

It may have dollar signs and other things that the shell might want to interpret.

Ah yes, correct Vanzin, I didn't think about that use case. So I will apply shellEscape for both in this PR.

Test build #89180 has finished for PR 21014 at commit 4732c4b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89180/
Test PASSed.

Hi, do I need to do something else in order for this PR to be merged ?

Wouldn't it be simpler and safer to just shellEscape everything in options before returning it?

Test build #89892 has started for PR 21014 at commit 2260e15.

Test build #89892 has finished for PR 21014 at commit 2260e15.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89892/
Test PASSed.

What I meant was to actually do options.map(shellEscape) in the return line.

Ah yes, I understand what you mean, I'm going to fix it right now

Test build #89896 has started for PR 21014 at commit 0038a23.

Test build #89896 has finished for PR 21014 at commit 0038a23.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89896/
Test PASSed.

Merging to master / 2.3 / 2.2.


New PubNub Instance on every Activity?

Hi,
Currently I'm working in the prototype of my own app for Android. I would like
use the PubNub services to do video streaming and calls, I just started to learn
how PubNub API and their services works.
I have been checking the example about AndroidRTC , and I noticed that on every activity where PubNub Services is required, a new instance is created therefore a new login everytime.
This is correct or this object should be in a singleton class?
and How I can do logout of PubNub?
In advance, Thanks!!!!

PubNub operates using a complex backend with caching of recently subscribed users and HTTP long polling. Messages are stored for a short amount of time so you will not miss anything if you transition between activities. Singleton isn't a bad idea for a messaging app where you should always be listening, but for AndroidRTC I was subscribing to different channels for different activities. Basically if I had a singleton, the procedure would be cancelling a HTTP request in unsubscribing from a channel and then reopening another for a different channel. I did the same thing allowing a slight overhead in recreating an object in exchange for channel manageability. I will make a singleton demo soon and let you know how it works.
Also since it is all polling, the "Logout" of pubnub is unsubscribing.
@Override
protected void onStop() {
     super.onStop();
     if(this.mPubNub!=null){
         this.mPubNub.unsubscribeAll();
     }
 }


add "gptqmodel_marlin_cuda_24" CUDAExtension

@ZX-ModelCloud  There are two marlin kernels in vllm. One is dense (the one that's compatible with GPTQModel) and spare(2-4) which uses/loads special 2-4 sparsified models.


Update known-issues.md

Update known issues page
One known issue gets resolved, remove the AAD issue from Serverless part (Active issue) to closed issues

Duplicate from
https://github.com/MicrosoftDocs/azure-docs-pr/pull/252703

Duplicate from
https://github.com/MicrosoftDocs/azure-docs-pr/pull/252703


Put a note in the feedback form that reminds mentees and mentors to complete feedback once a week

Put it on the main page where it is more visible for users

This shouldn't be in Review/QA. It hasn't been done yet


an Error

Thank you for your great work, but when I try to train the model using the command "python -u src/run_kbc_subgraph.py --dataset conceptnet --evaluate-every 10 --n-layers 2 --graph-batch-size 60000 --sim_relations --bert_concat".
I got an error as follow:
Traceback (most recent call last):
File "src/run_kbc_subgraph.py", line 532, in
main(args)
File "src/run_kbc_subgraph.py", line 349, in main
loss = model.get_score(e1, rel, target, graph_embeddings)
File "/home/s2020017/work/commonsenseKBC/src/model.py", line 155, in get_score
reg_loss = self.regularization_loss(embedding)
File "/home/s2020017/work/commonsenseKBC/src/model.py", line 134, in regularization_loss
dec_weight = self.decoder.module.w_relation.weight.pow(2)
File "/home/s2020017/Anaconda_3.7/envs/commonsenseKBC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 518, in getattr
type(self).name, name))
AttributeError: 'ConvTransE' object has no attribute 'module'
How can I fix it?
Thank you so much!!

Thank you for your great work, but when I try to train the model using the command "python -u src/run_kbc_subgraph.py --dataset conceptnet --evaluate-every 10 --n-layers 2 --graph-batch-size 60000 --sim_relations --bert_concat".
I got an error as follow:
Traceback (most recent call last):
File "src/run_kbc_subgraph.py", line 532, in
main(args)
File "src/run_kbc_subgraph.py", line 349, in main
loss = model.get_score(e1, rel, target, graph_embeddings)
File "/home/s2020017/work/commonsenseKBC/src/model.py", line 155, in get_score
reg_loss = self.regularization_loss(embedding)
File "/home/s2020017/work/commonsenseKBC/src/model.py", line 134, in regularization_loss
dec_weight = self.decoder.module.w_relation.weight.pow(2)
File "/home/s2020017/Anaconda_3.7/envs/commonsenseKBC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 518, in getattr
type(self).name, name))
AttributeError: 'ConvTransE' object has no attribute 'module'
How can I fix it?
Thank you so much!!

You can use the ' dec_weight = self.decoder.w_relation.weight.pow(2)' instead of ' dec_weight = self.decoder.module.w_relation.weight.pow(2)'


Redirect to Login page if not logged in currently

My goal is to force a log in for all users in order to access the knowledge base in any way. Currently, it seems that the configuration properties for authentication shows or hides the Log In button, providing the option to log in. This configuration does not require a log in for viewing no matter what combination of config auth settings are made.
I did find the following workaround, which seems to work on my dev machine to force the user to the Log In page on any page load.
Added the following lines to the Layout.html file between the head and body DOM elements.
{{^loggedIn}}
{{#config.googleoauth}}
<script> window.location = "{{config.base_url}}/auth/login";</script>
{{/config.googleoauth}}
{{^config.googleoauth}}
<script> window.location = "{{config.base_url}}/login";</script>
{{/config.googleoauth}}
{{/loggedIn}}
This works because Layout.html is the full layout for all of the other partial layouts templates, except for Login.html which is it's own full layout. So far I have not seen any AJAX content loading in raneto, it relies instead on page loads so this code will fire on any content request other than the Login page.

@thomaspurk please check out @mohammadrafigh 's contribution
You'll have to clone the repo (master branch) and we'll publish to NPM soon

@ryanlelek does rane support win10? i have error log in,could you please help me?
this is logfiles
GET /login 200 5.976 ms - 2836                                                  GET /lib/jquery/dist/jquery.min.js 403 3.259 ms - 32                            GET /styles/login-style.css 304 5.095 ms - -                                    GET /lib/bootstrap/dist/js/bootstrap.min.js 403 5.882 ms - 32                   GET /styles/zocial.css 304 6.131 ms - -                                         GET /lib/sweetalert2/dist/sweetalert2.all.min.js 403 6.775 ms - 32              GET /lib/jquery-backstretch/jquery.backstretch.min.js 403 2.101 ms - 32         GET /lib/bootstrap/dist/css/bootstrap.min.css 403 1.557 ms - 32                 GET /styles/raneto.css 304 1.692 ms - -                                         GET /styles/login-form.css 304 1.721 ms - -                                     GET /scripts/login.js 304 2.297 ms - -                                          Error: Whoops. Looks like this page doesn't exist.                              Error: Whoops. Looks like this page doesn't exist.                              Error: Whoops. Looks like this page doesn't exist.                              Error: Whoops. Looks like this page doesn't exist.                              Error: Whoops. Looks like this page doesn't exist.                              POST /login 404 14.233 ms - 145


Firefox support

It would be really nice to have this extension on Firefox. I did some testing and it appears that the current version will work without any code modifications. The only thing to do is package the extension and publish it to the add-on repository.

Good idea, I submitted it to the Firefox store, it might take a few weeks to get reviewed. I will keep you updated.

Ok it seems to be approved and live on the store https://addons.mozilla.org/en-US/firefox/addon/hello-new-tab/
It works on my PC but let me know if there are any issues


Can i use this on macos? Specifically, on an ARM (M1) based mac?

It's entirely possible, and has been tested in termix. Would not advise doing so with larger models, as this can be problematic. I've joined the KoboldAI team and we are collaborating on that now, it has much better handling of all this than Avril does.


Friendly url issue

we have friendly ids like '41-hello', there is no way to convert and validate with rails_param.

my work around is
param! :program_id, String, required: true, transform: :to_i

but would be nice to have a built in solution for such common usecase

I'm not really sure what you were trying to achieve this, maybe you want to use format?
Check out the last example to see it's usage: https://github.com/nicolasblanco/rails_param#example
Closing for now as really hold, but please feel free to reopen if needed


Workflow execution

Minimal workflow execution schema.
Implementation at https://github.com/common-workflow-language/cwltool-service/pull/1

@tetron I believe @briandoconnor had been poking at a branch off of what you had a few days ago based on a convo he & i had. we will likely need to coordinate.

Sure, this is just intended as a starting point for the discussion.

hey @tetron  and @geoffjentry I'm working on a branch from Peter I forked off like a week ago.  Let me merge this branch into master, get my branch in shape, and then make a new pull request.  Any objections?  There are a few open pull requests from several months back, not sure how to best incorporate this work.  But I think it's important to get WES going and in sync with the Task Execution Service


Consider renaming METdatadb to METdataio or something similar

Describe the Task
The METdatadb package is being update for the METplus 4.1.0 coordinated release to include expanded I/O support of various input file formats. However, the existing METdatadb name implies support only for database read/write access.
For the next major coordinated release (METplus 5.0.0), consider renaming this package as METdataio, or something similar. This would involve several steps and would include changes to:

the name of this repository
access to this package from METplus, METviewer, METexpress, METcalcpy, and METplotpy
updates to the documentation
updates to the website/training materials, including an updated logo

Time Estimate
Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.
Sub-Issues
Consider breaking the task down into sub-issues.

[ ] Add a checkbox for each sub-issue here.

Relevant Deadlines
List relevant project deadlines here or state NONE.
Funding Source
Define the source of funding and account keys here or state NONE.
Define the Metadata
Assignee

[x] Select engineer(s) or no engineer required: @venitahagerty
[ ] Select scientist(s) or no scientist required

Labels

[x] Select component(s)
[x] Select priority
[x] Select requestor(s)

Projects and Milestone

[ ] Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
[ ] Select Milestone as the next official version or Future Versions

Define Related Issue(s)
Consider the impact to the other METplus components.

[x] METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy
If/when we proceed with this change, define GitHub issues to accommodate it in METplus, METviewer, METexpress, METcalcpy, and METplotpy.

Task Checklist
See the METplus Workflow for details.

[ ] Complete the issue definition above, including the Time Estimate and Funding Source.
[ ] Fork this repository or create a branch of develop.
Branch name: feature_<Issue Number>_<Description>
[ ] Complete the development and test your changes.
[ ] Add/update log messages for easier debugging.
[ ] Add/update unit tests.
[ ] Add/update documentation.
[ ] Push local changes to GitHub.
[ ] Submit a pull request to merge into develop.
Pull request: feature <Issue Number> <Description>
[ ] Define the pull request metadata, as permissions allow.
Select: Reviewer(s) and Linked issues
Select: Repository level development cycle Project for the next official release
Select: Milestone as the next official version
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Close this issue.

Changes have been made.


add base/centos8 image

This PR adds a new base image for centos8

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Going to close this image as it's over a year old. If you still need this image you can generate a new one using one of these scripts: https://github.com/cypress-io/cypress-docker-images/blob/master/package.json#L12-L14


update to 0.1.20 error running the production script

I upgraded to 0.1.20 to use the stylesheet optional paramenters.
Now when I run in production (that it was perfectly working before), I have this error:
          | /usr/src/app/webpack/webpack.config.client.production.js:105
          | scss_loader.loader = _extractTextWebpackPlugin2.default.extract(scss_loader.loaders.shift(), scss_loader.loaders.join('!'));
          |                                                                                ^
          |
          | TypeError: Cannot read property 'shift' of undefined
          |     at Object.<anonymous> (webpack.config.client.production.js:83:50)
          |     at Module._compile (module.js:409:26)
          |     at loader (/usr/src/app/node_modules/babel-register/lib/node.js:158:5)
          |     at Object.require.extensions.(anonymous function) [as .js] (/usr/src/app/node_modules/babel-register/lib/node.js:168:7)
          |     at Module.load (module.js:343:32)
          |     at Function.Module._load (module.js:300:12)
          |     at Module.require (module.js:353:17)
          |     at require (internal/module.js:12:17)
          |     at Object.<anonymous> (/usr/src/app/webpack/webpack.config.client.production.entry.js:3:18)
          |     at Module._compile (module.js:409:26)
          |     at Object.Module._extensions..js (module.js:416:10)
          |     at Module.load (module.js:343:32)
          |     at Function.Module._load (module.js:300:12)
          |     at Module.require (module.js:353:17)
          |     at require (internal/module.js:12:17)
          |     at module.exports (/usr/src/app/node_modules/webpack/bin/convert-argv.js:80:13)
          |
          | npm info react-frost@ Failed to exec production-build-client script
          | npm ERR! Linux 4.4.0-31-generic
          | npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run-script" "production-build-client"
          | npm ERR! node v4.4.7
          | npm ERR! npm  v2.15.8
          | npm ERR! code ELIFECYCLE
          | npm ERR! react-frost@ production-build-client: `webpack --colors --display-error-details --config ./webpack/webpack.config.client.production.entry.js`
          | npm ERR! Exit status 1
          | npm ERR!
          | npm ERR! Failed at the react-frost@ production-build-client script 'webpack --colors --display-error-details --config ./webpack/webpack.config.client.production.entry.js'.
          | npm ERR! This is most likely a problem with the react-frost package,
          | npm ERR! not with npm itself.
          | npm ERR! Tell the author that this fails on your system:
          | npm ERR!     webpack --colors --display-error-details --config ./webpack/webpack.config.client.production.entry.js
          | npm ERR! You can get information on how to open an issue for this project with:
          | npm ERR!     npm bugs react-frost
          | npm ERR! Or if that isn't available, you can get their info via:
          | npm ERR!
          | npm ERR!     npm owner ls react-frost
          | npm ERR! There is likely additional logging output above.
          |
          | npm ERR! Please include the following file with any support request:
          | npm ERR!     /usr/src/app/npm-debug.log
          | ERROR: "production-build-client" exited with 1.
          |
          | npm info react-frost@ Failed to exec production script
          | npm ERR! Linux 4.4.0-31-generic
          | npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "production"
          | npm ERR! node v4.4.7
          | npm ERR! npm  v2.15.8
          | npm ERR! code ELIFECYCLE
          | npm ERR! react-frost@ production: `npm-run-all production-build-client production-build-server production-server`
          | npm ERR! Exit status 1
          | npm ERR!
          | npm ERR! Failed at the react-frost@ production script 'npm-run-all production-build-client production-build-server production-server'.
          | npm ERR! This is most likely a problem with the react-frost package,
          | npm ERR! not with npm itself.
          | npm ERR! Tell the author that this fails on your system:
          | npm ERR!     npm-run-all production-build-client production-build-server production-server
          | npm ERR! You can get information on how to open an issue for this project with:
          | npm ERR!     npm bugs react-frost
          | npm ERR! Or if that isn't available, you can get their info via:
          | npm ERR!
          | npm ERR!     npm owner ls react-frost
          | npm ERR! There is likely additional logging output above.
          |
          | npm ERR! Please include the following file with any support request:
          | npm ERR!     /usr/src/app/npm-debug.log

it seems that the loaders property undefined.
Do you think that it is connected to your last modifications?

Post your  /usr/src/app/webpack/webpack.config.client.production.js and I'll see

import language from './language'

import path     from 'path'

import webpack             from 'webpack'
import configuration  from './webpack.config.client'
import clean_plugin        from 'clean-webpack-plugin'
import extract_text_plugin from 'extract-text-webpack-plugin'

// const configuration = Object.clone(base_configuration)

// const server_output_path = path.resolve(configuration.output.path, '../server')

configuration.devtool = 'source-map'

configuration.plugins = configuration.plugins.concat(
  // clears the output folder
  new clean_plugin([path.relative(configuration.context, configuration.output.path)], { root: configuration.context }),

  // environment variables
  new webpack.DefinePlugin ({
    'process.env': {
      // Useful to reduce the size of client-side libraries, e.g. react
      NODE_ENV: JSON.stringify('production') // 'development' to see non-minified React errors
    },

    _development_       : false,
    _production_        : true,
    _development_tools_ : false  // enable/disable redux-devtools
  }),

  // css files from the extract-text-plugin loader
  // (for more information see the extract_text_plugin code for scss_loader below)
  //
  // the "allChunks: true" option means that this extracted file will contain
  // the styles from all chunks of an entry (each entry can be divided into chunks).
  // without this option styles would only be extracted from the top-level chunk of an entry.
  new extract_text_plugin('[name]-[contenthash].css', { allChunks: true }),

  // omit duplicate modules
  new webpack.optimize.DedupePlugin(),

  // Assign the module and chunk ids by occurrence count.
  // Ids that are used often get lower (shorter) ids.
  // This make ids predictable, reduces to total file size and is recommended.
  new webpack.optimize.OccurenceOrderPlugin(),

  // extracts common javascript into a separate file (works)
  // new webpack.optimize.CommonsChunkPlugin('common', 'common.[hash].js'),

  // Compresses javascript files
  new webpack.optimize.UglifyJsPlugin({
    compress: {
      warnings: false
    }
  })
)

// // don't know why they write it like this
// configuration.output.filename = '[name]-[chunkhash].js'

// network path for static files: fetch all statics from the static server
// configuration.output.publicPath = `http://localhost:3001/build/`

// begin: set extract text plugin as a Css loader

// find the styles loader
const scss_loader = configuration.module.loaders.filter(loader => {
  return loader.test.toString() === configuration.regular_expressions.styles.toString()
})[0]

// https://github.com/webpack/extract-text-webpack-plugin
//
// It moves every require("style.css") in entry chunks into a separate css output file.
// So your styles are no longer inlined into the javascript, but separate
// in a css bundle file (styles.css). If your total stylesheet volume is big,
// it will be faster because the stylesheet bundle is loaded in parallel to the javascript bundle.
// (but it also disables hot module reload)
//
// the first argument to the .extract() function is the name of the loader
// ("style-loader" in this case) to be applied to non-top-level-chunks in case of "allChunks: false" option.
// since in this configuration "allChunks: true" option is used, this first argument is irrelevant.
scss_loader.loader = extract_text_plugin.extract(scss_loader.loaders.shift(), scss_loader.loaders.join('!'))
delete scss_loader.loaders

// done: set extract text plugin as a Css loader

export default configuration

if you notice I commented the configuration clone line..

Well, the error means that scss_loader wasn't found.
Try to do `console.log(require('util').inspect(configuration.module.loaders))

indeed I don't have a loaders but a loader..
[ { test: /\.json$/, loader: 'json-loader' },
             { test: /\.js$/,
               exclude: /node_modules/,
               loader: 'babel-loader' },
             { test: /\.scss$/,
               loader: 'style-loader!/usr/src/app/node_modules/extract-text-webpack-plugin/loader.js?{"omit":1,"extract":true,"remove":false,"id":1}!style!css?sourceMap!resolve-url?sourceMap!postcss-loader!sass-loader?outputStyle=expanded&sourceMap=true&sourceMapContents=true' },
             { test: /\.svg/,
                loader: 'url-loader?limit=26000&mimetype=image/svg+xml' },
             { test: /\.(jpg|png)$/, loaders: [ 'url-loader?limit=10000' ] },
             { test: /\.rt/,
               loader: 'react-templates-loader',
               include: '/usr/src/app/webpack/application' } ]

So, what you have there is
 loader: 'style-loader!/usr/src/app/node_modules/extract-text-webpack-plugin/loader.js?{"omit":1,"extract":true,"remove":false,"id":1}!style!css?sourceMap!resolve-url?sourceMap!postcss-loader!sass-loader?outputStyle=expanded&sourceMap=true&sourceMapContents=true' },

This reminds me of the "flash of unstyled content" issue solution.
Why do you have it enabled in production?

my webpack.config.client is indeed
import { client_configuration } from 'universal-webpack'
import settings from './universal-webpack-settings'
import configuration from './webpack.config'

export default client_configuration(configuration, settings, { development: true, css_bundle: true })

I just turn it on for development and I didn't figure out that I was using the same file for production as well.
It works!

Also, I thought that that boilerplate code with extract-text-webpack-plugin is common enough to include it into the library as well.
In the latest version of universal-webpack it's now possible to pass development: false parameter and extract-text-webpack-plugin will be applied automatically (unless css_bundle is set to false).
I guess everyone does that in production.
The relevant commit to the example app:
https://github.com/halt-hammerzeit/webpack-react-redux-isomorphic-render-example/commit/807c76d872dff1687f097bae2c201aa6ef91f0dd

That is much better. Good Idea!


Help, error - (25)

Please help bot displays error:
Error: There was an error accepting this trade offer.  Please try again later. (25)

#121
Also,
LimitExceeded = 25;

You have too many trade offers, either total or too one person.

sorry


Docs(about): fix typo and remove code non used

Thanks for the heads up! This was addressed in another branch updating the bios.


Linear Sort Algorithm?

Any way the linear search algorithm can be added to this repo?
You can use the three types of linear search:

linearSearch
betterLinearSearch
sentinalLinearSearch

There is a good implementation of this in type script found at the following repo:
algorithms-unlocked-linear-search

Can you do PR with your change?

Is Any PR done ?

I do have a PR for Sentinal Linear Search Algorithm. Can anyone give me access to raise a PR?

@bf4648 Can u     assign me to do this


Adjust scraping logic to current reality

HTML page changed a lot.
Use different selectors, get rid of a


ack discarded messages

ack discarded messages, so they won't get processed again

/cc @CrystalChun


Setting other port than 10502 in Modbus Server node does not work

The Modbus Server node constantly uses port 10502 (also response delay is fixed) for its communication independent what is configure in the node's configuration.
Relevant source code line in modbus-server-js for this problem :
}).refs({'logEnabled': true, 'port': 10502, 'responseDelay': 100})

please test it again with v0.9.1


Can I change Admin labels?

When I add a key to the "/cms/index.js" object, it goes to the admin with that exact key name. Is there any way I can customize this?
Let's say my key is "firstName" but I want to display it as "Your Name". Is this possible?
Thanks for the great work.

It appears this issue should be in the enduro admin repo. Specifically where crumbs are located is here inside the angular app portion of the admin: https://github.com/Gottwik/enduro_admin/blob/d574491f4ec4bdc9ec0cc823cfb0c2d4c93c719a/assets/js/views/cms_edit.html#L18

This issue is related to the admin interface, I've gone ahead and opened a new issue referenced above since I'm interested in this as well.

Thanks @rnlambert7, I'm closing this one

sure, just do:
{
	$first_name_label: 'What\'s your name',
	first_name: 'Kiko',
}

this is a standard feature, just do:
{
	$first_name_label: 'What\'s your name',
	first_name: 'Kiko',
}


update to use CSBase

Is waiting for

https://github.com/JuliaRegistries/General/pull/67921

Codecov Report

Merging #59 (88c19be) into master (24fbe05) will decrease coverage by 0.34%.
The diff coverage is 73.41%.

@@            Coverage Diff             @@
##           master      #59      +/-   ##
==========================================
- Coverage   89.98%   89.63%   -0.35%
==========================================
  Files          17       17
  Lines        2525     2527       +2
==========================================
- Hits         2272     2265       -7
- Misses        253      262       +9

Impacted Files
Coverage Δ

src/mimo_diskmargin.jl
97.08% <ø> (ø)

src/reduction.jl
92.68% <0.00%> (ø)

src/hinfinity_design.jl
88.58% <26.66%> (ø)

src/uncertainty_interface.jl
83.21% <50.00%> (-2.99%)
:arrow_down:

src/lqg.jl
75.15% <57.14%> (ø)

src/ExtendedStateSpace.jl
86.69% <90.00%> (ø)

src/canonical.jl
97.18% <100.00%> (ø)

src/descriptor.jl
94.73% <100.00%> (ø)

src/diskmargin.jl
95.14% <100.00%> (ø)

src/find_lft.jl
85.29% <100.00%> (ø)

... and 6 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


How to download 773 cases with pseudo tumor labels without Baidu account

First of all thanks for your effort in making this incredible contribution to medical analysts! However I cannot find a way to download the 773 cases with pseudo tumor labels without Baidu account (I'm in Austria, with an Austrian mobile number) .. is there a Dropbox/MEGA/alternative link available?
Thanks!

Hi @sirfoga ,
Thanks for your interest.
I used dropbox but it is not stable.
I just added Google drive download link. Please let me know if you have any further problems.
Best,
Jun


Add Color Picker

Add a color picker for any devices that have RGB channels (right now they have to be named "red", "green", and "blue") so the stairville-led-par-56, lixda-par12-led, eurolite-led-tha-120PC, briteq-bt-theatre-60FC, and (if #51 is accepted) those two devices will work. Others won't right now.

Uh that is an interesting UX in macOS - today I learned about input type = 'color' :)
Not the nicest UI imho, but it fits in perfectly to the current state of the rest 😆 - perfect for the first iteration thanks 👍

Can't really control the UI that various OSes/browsers put on it. The phone one in particular is really bizarre. But it does open the door for a nicer UI addition later.

Yeah <input type="color"> looks different in every browser or OS and is often unsupported, but since the UI is not great anyway, it is good enough to make it like this. 😉

But it does open the door for a nicer UI addition later.

There are people and projects working on a React client, which only uses node-dmx as the backend API.
node-dmx is a great library with a very functional oldschool UI. I wouldn't stop anyone, but I don't see the benefit in improving that UI, it's better to build new clients with modern web development techniques.


does http_send() trigger cloudflare protections?

I'm fetching the Arts & Letters Daily front page but getting 403-ed when using http_send. It works fine though when using native Python and with curl (indicating that it's neither an IP nor a headlessness issue). It almost looks like Cloudflare's anti-bot protections are triggered with one and not the other for some reason.
Inspecting the network packets showed that http_send will lowercase and reorder the headers, which I was told is how Rust's HeaderMap works. While this is perfectly fine by the specs I was worried that it might be used to sanity-check the user-agent string I'm habitually spoofing. (Firefox for example will always send the host header first and in title-case.) Before turning to the discord chat this was my best guess at explaining why with some sites I've been getting different responses depending on whether I fetch natively or spinned but the friendly folks over at the discord server have quickly figured out that ordering and case apparently don't matter. (And indeed, it turns out that any user-agent string at all will work as long as one is specified.) Instead I was pointed to the phenomenon of TLS fingerprinting as a more likely possibility.
Specifically:
from urllib.request import urlopen, Request

def fetch():
	req = Request("https://aldaily.com", headers = { "user-agent": "X" })
	res = urlopen(req)
	return res.read().decode("utf8") if res.status == 200 else None

works fine, but
from spin_http import http_send, Request

def fetch():
	req = Request("GET", "https://aldaily.com", { "user-agent": "X" }, None)
	res = http_send(req)
	return res.body.decode("utf8") if res.status == 200 else None

from the same IP would fail with HttpError::DestinationNotAllowed. One basically gets the same response as with a naked curl call:
curl 'https://www.aldaily.com'

By contrast, the simplest working curl call is:
curl 'https://www.aldaily.com' -H 'User-Agent: X'

Hey @dejbug,
That error you are running into seems to be due to not setting a allowed_http_hosts value in your spin.toml.
[[component]]
.
.
allowed_http_hosts = ["https://www.aldaily.com"]

Also, note that you need https://www.aldaily.com as https://aldaily.com has a permanent redirect.
reference: https://developer.fermyon.com/spin/writing-apps#granting-networking-permissions-to-components

Hi @karthik2804 !
Hmm. My spin.toml looks like this (so far):
# ...

[[component]]
id = "ibtimes"
source = "ibtimes.wasm"
environment = { RATE = "3600" }
key_value_stores = ["default"]
allowed_http_hosts = [ "https://www.ibtimes.com" ]

[component.trigger]
route = "/ibtimes"

[component.build]
command = "spin py2wasm ibtimes -o ibtimes.wasm"
watch = ["ibtimes.py", "Store.py", "fetch.py", "abs.py", "Pipfile"]

[[component]]
id = "aldaily"
source = "aldaily.wasm"
environment = { RATE = "3600" }
key_value_stores = ["default"]
allowed_http_hosts = [ "https://www.aldaily.com" ]

[component.trigger]
route = "/aldaily"

[component.build]
command = "spin py2wasm aldaily -o aldaily.wasm"
watch = ["aldaily.py", "Store.py", "fetch.py", "Pipfile"]

[[component]]
id = "weforum"
source = "weforum.wasm"
environment = { RATE = "3600" }
key_value_stores = ["default"]
allowed_http_hosts = [ "https://www.weforum.org" ]

[component.trigger]
route = "/weforum"

[component.build]
command = "spin py2wasm weforum -o weforum.wasm"
watch = ["weforum.py", "Store.py", "fetch.py", "Pipfile"]

Hey @dejbug in your python code, can you change the request URL to be https://www.aldaily.com.
Alternatively in the spin.toml you can add https://aldaily.com in addition to the existing (i.e)
allowed_http_hosts = [ "https://aldaily.com", "https://www.aldaily.com" ]

This is the piece of code I am using and it seems to be returning the webpage properly
from spin_http import Request, Response, http_send

def fetch():
        req = Request("GET", "https://www.aldaily.com", { "user-agent": "X" }, None)
        res = http_send(req)
        return res.body.decode("utf8") if res.status == 200 else None

def handle_request(request):
    print(fetch())
    return Response(200, {"content-type": "text/plain"})

and here is my spin.toml
spin_manifest_version = "1"
authors = ["karthik2804<EMAIL_ADDRESS>description = ""
name = "test-http"
trigger = { type = "http", base = "/" }
version = "0.1.0"

[[component]]
id = "test-http"
source = "app.wasm"
allowed_http_hosts = ["https://www.aldaily.com"]
[component.trigger]
route = "/..."
[component.build]
command = "spin py2wasm app -o app.wasm"
watch = ["app.py", "Pipfile"]

You're right @karthik2804 ! I've just tried it in a freshly spinned-up, single-component app like yours and it works as expected. At least this proves that it's not an issue with http_send. Now I need to figure out what is going wrong in my multi-component attempt.
Thanks a bunch! and please feel free to close this issue.

Hmm... unless it's a compounded issue? Maybe http_send() in conjunction with my specific IP (I'm in Serbia right now, which seems to trigger some occasional Cloudflares while browsing normally)? Do you think this possible? Is there an easy way to test? Maybe with a VPN?

Amazing.
When I use an agent string of Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0 I get the CAPTCHA. When I use an agent string of X I get the proper page. Both work with curl but the first one makes http_send fail. I hope I'm hallucinating, it's late.

I am not entirely sure what is going on there. I do not have any particularly good guesses either.

@dejbug I am closing this now but feel free to reopen if there are more questions.


[BUG] Disk shows replicas that don't exist. Node shows none.

Describe the bug (🐛 if you encounter this issue)
A clear and concise description of what the bug is.
Node shows 0 replicas, but is over-allocated (was 100% allocated with 0% used). Disk shows 104 replicas, but has an empty list.
To Reproduce
Steps to reproduce the behavior:

Expected behavior
Node shows total number of replicas on all disks. Replicas are properly listed.
Log or Support bundle
supportbundle_a33949f1-e247-4a26-a6cd-3dcac7397190_2023-03-16T13-46-44Z.zip
Environment

Longhorn version: v1.41
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s

Number of management node in the cluster: 1
Number of worker node in the cluster: 3

Node config

OS type and version: Rasbian 64bit (Debian 11)
CPU per node: 4
Memory per node: 4GB
Disk type(e.g. SSD/NVMe): USB 3.0 Flash Drive
Network bandwidth between the nodes: Gigabit

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
Number of Longhorn volumes in the cluster: 11

Additional context
One of my nodes randomly filled up, and won't clean itself up. I noticed then when I was having constant rebuild failures on volumes. I was trying to remove and re-add the disk since the directory was empty, but the disk in the UI shows 104 replicas (but no listed ones), and was at 100% allocated until I changed it to no longer overprovision, and is now at 187% allocation. Eviction doesn't do anything.

I resolved this by unmounting the disk from the host, removing it from longhorn, and re-adding it. The space was available again, but I'm having a new issue where all the volumes on this host are stopped no matter what I do. I'm not sure if I should file a new big for that or not.

I think this is because you unmouned the disk from the host, removed it from longhorn, and re-added it before detaching it, right?
Can the volumes be reattach the volumes?

Sorry, what do you mean? The "new" disk seems healthy and fine, and pods running on this node can attach to volumes, but replicas are all showing as stopped.

I'll open a new bug with a new bundle

I also encountered this I believe as a result of longhorn trying to replica volumes while my cluster was unstable. I've managed to get to the same point of the replicas being gone but new replicas always being stopped. Here's the method I used:

Evict all working volumes and disable scheduling. I did this at both node and disk level.
Navigate to the directory for the disk and delete all files. This includes the config file and the replicas. Longhorn UI should update automatically but the values won't make sense.
Edit the node to delete the disk. This should now be possible as longhorn detects no replicas on the disk.
Recreate the disk. If you get an error about the node syncing, just wait a few seconds and try again.
Re-enable the node.

I'm now looking at #5762 to try resolve the stopped replicas issue.


‘Renew after’ is in the future, but my cert expired today

I have a certificate on IIS which is due to expire today, but am getting the following message:
Checking IIS website.com (C:\Temp) Renew After 07/05/2016
I now have this issue on two different servers, one running a slightly older version of letsencrypt-win-simple and one running the latest version compiled by myself in VS.
Thanks

Did you use the manual plugin or the IIS plugin when you first created the certs?

I created the initial certificate using the executable/command line, which also created the Windows scheduled task for me. I then ran the command that you can see in the task scheduler manually, which gave me the notice about the certificate only being renewable after 07/05/2016, which is well after the cert has expired.

How did you run the app when you created the certificate the first time?

I  just ran letsencrypt.exe from the command line and it took me through
the process, asked me to choose an IIS site and so on.
On 5 April 2016 at 17:12, Robert Kerber<EMAIL_ADDRESS>wrote:

How did you run the app when you created the certificate the first time?
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/Lone-Coder/letsencrypt-win-simple/issues/182#issuecomment-205876570

Ok, then that means you used the IIS plugin and not the Manual plugin. That's good because the Manual plugin doesn't auto-renew.
If you are using V 1.8 or greater check your event logs for any errors or warnings logged by letsencrypt_win_simple to the Application log. If you aren't, there is no logging so not much we can do to troubleshoot it.
If it had any errors or warnings while trying to renew, they should be in your event log.
You can also change the minimum log level to Debug and try to run the renew again.

I've attached an image of what I'm getting.
Thanks.
On 5 April 2016 at 17:59, Robert Kerber<EMAIL_ADDRESS>wrote:

Ok, then that means you used the IIS plugin and not the Manual plugin.
That's good because the Manual plugin doesn't auto-renew.
If you are using V 1.8 or greater check your event logs for any errors or
warnings logged by letsencrypt_win_simple to the Application log. If you
aren't, there is no logging so not much we can do to troubleshoot it.
If it had any errors or warnings while trying to renew, they should be in
your event log.
You can also change the minimum log level to Debug
https://github.com/Lone-Coder/letsencrypt-win-simple/wiki/Application-Logging#change-log-level
and try to run the renew again.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/Lone-Coder/letsencrypt-win-simple/issues/182#issuecomment-205894924

Hmm, image didn't attach. Here it is:
http://u.sv1.uk/uploads/o_1afjp8ufocv015uq1bfv13b4ecga.PNG

Now since the renewal isn't until the 22nd, it won't renew it until then.
If you open regedit and go to HKCU\SOFTWARE\letsencrypt-win-simple\https://acme-staging.api.letsencrypt.org/, edit the Renewals and change the renewal date to be in the past.
Then you can rerun the renewal.

Oh ok, so it didn't update that reg key properly for some reason.
My cert expired yesterday now, so that date is a bit out for some reason. The same happened on my own server and another server I manage.
Changing the date in that key worked perfectly and it seems to have set the correct 'renew after' date now.
Thank you for your help :)


Update to version 2.5.x - APK 0.53.1 - IPA 1.23.1

The APK has been updated to version 0.53.1

http://www.apkmirror.com/apk/niantic-inc/pokemon-go/pokemon-go-0-53-1-release/pokemon-go-0-53-1-android-apk-download/

Recent Datamine

https://thesilphroad.com/news/pokemon-go-v053-apk-mine-new-moves-and-evolution-items
https://www.reddit.com/r/TheSilphRoad/comments/5okrcw/the_silph_roads_apk_mine_of_v0531_is_complete_38/

Metadata dump and more data mining efforts:

https://gist.github.com/apavlinovic/c326381e13909bd0ff3b3ec4a17642e7

http://pokemongohub.net/0-53-1-data-mining-unown-added-new-avatar-customization-options/


Sometimes, the end day setting in traderclient account can't be same with the end day of get_bars().

Describe the bug
The end day setting in traderclient account have to be later than the end day of get_bars().
To Reproduce
Steps to reproduce the behavior:

setting same end day in traderclient account and  get_bars()
doing backtest traders

Expected behavior
I hope they could be set the same day. Otherwsie, it has impact on my metrics result.
Screenshots

RemoteProtocolError                       Traceback (most recent call last)
File c:\Users\Admin\anaconda3v2\envs\omicron\lib\site-packages\httpx_exceptions.py:326, in map_exceptions(mapping, **kwargs)
[325](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_exceptions.py?line=324) try:
--> [326](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_exceptions.py?line=325)     yield
[327](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_exceptions.py?line=326) except Exception as exc:
File c:\Users\Admin\anaconda3v2\envs\omicron\lib\site-packages\httpx_client.py:861, in Client._send_single_request(self, request, timeout)
[860](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_client.py?line=859) with map_exceptions(HTTPCORE_EXC_MAP, request=request):
--> [861](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_client.py?line=860)     (status_code, headers, stream, ext) = transport.request(
[862](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_client.py?line=861)         request.method.encode(),
[863](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_client.py?line=862)         request.url.raw,
[864](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_client.py?line=863)         headers=request.headers.raw,
[865](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_client.py?line=864)         stream=request.stream,  # type: ignore
[866](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_client.py?line=865)         ext={"timeout": timeout.as_dict()},
[867](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_client.py?line=866)     )
[869](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_client.py?line=868) def on_close(response: Response) -> None:
File c:\Users\Admin\anaconda3v2\envs\omicron\lib\site-packages\httpcore_sync\connection_pool.py:218, in SyncConnectionPool.request(self, method, url, headers, stream, ext)
[217](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpcore/_sync/connection_pool.py?line=216) try:
--> [218](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpcore/_sync/connection_pool.py?line=217)     response = connection.request(
[219](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpcore/_sync/connection_pool.py?line=218)         method, url, headers=headers, stream=stream, ext=ext
[220](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpcore/_sync/connection_pool.py?line=219)     )
[221](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpcore/_sync/connection_pool.py?line=220) except NewConnectionRequired:
...
[340](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_exceptions.py?line=339)     raise
[342](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_exceptions.py?line=341) message = str(exc)
--> [343](file:///c%3A/Users/Admin/anaconda3v2/envs/omicron/lib/site-packages/httpx/_exceptions.py?line=342) raise mapped_exc(message, **kwargs) from exc
RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
Version
backtest version:  traderclient == 0.3.9, backtesting==0.4.8.

closed as it's a misusing error.


Fix cflib parameter handling

This should close #234 but not yet tested.
@whoenig, if you have the time to test it out together with the fancy new mocap functionality?

Both work fine:-) I tested the hello_world example in our mocap with a single marker. Ship it!
(code looks good, too!)

I did have to install additional libraries:
sudo apt install ros-humble-tf-transformations
sudo pip3 install transforms3d

not sure if we have that in the documentation already.

Awesome! thanks.
The installation instructions for the extra pip package are there already here: https://imrclab.github.io/crazyswarm2/installation.html


opentelemetry-collector: init at 0.26.0

Motivation for this change
Add https://github.com/open-telemetry/opentelemetry-collector/.
Note this is built using the source at https://github.com/open-telemetry/opentelemetry-collector-contrib/ rather than the main project, as it has more receivers and exporters. It has the same license and is a strict superset as noted in the description:

Please note that this repository and its releases are a superset of the core repository.

However I have left the binary name as otelcontribcol.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[x] macOS
[x] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 123232 at 43975052 run on x86_64-linux 1

1 package failed to build:

opentelemetry-collector

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

@ofborg build opentelemetry-collector opentelemetry-collector.passthru.tests

lukegb@Mac-mini pr123232 % ./result/bin/otelcontribcol --version
otelcontribcol version latest


[FEATURE] Allow for messages from NSFW channels to be posted to non-NSFW starboards, via a configuration option

Is your feature request related to a problem? Please describe.
Currently, the bot will not recognise messages from NSFW channels as being necessary of posting to a starboard, if the message's origin channel is NSFW and the starboard for posting is non-NSFW.
Describe the solution you'd like
I believe there should be a config option (such as via /starboards edit behaviour [starboard:] allow-nsfw-origin: [True/False]) to allow such behaviour.
Describe alternatives you've considered
The only working alternative is to enable NSFW on the starboard channel, which is unideal for our use case. Another alternative would be to use the /utils force [message-link:] command however, as mentioned in issue #113, that command also suffers from the same issue.

I'll probably do this, going to think about it first though.
A forced message should override NSFW-protection now.

This won't be done


Slow bulkload performance due to high GC for couple of hundred million rows

Added an overloaded function, which requires that sequence of headers and sequence of columns are in the sorted order as required by HBase (i.e. sorted by columns names) . This avoids creation of TreeSet objects to sort the columns for every row. It also avoids creation of the Map objects for each row where there is mapping of a column name to column value for each row.
It does reduce the GC time in our case with ~ 800 million rows with 100 HBase regions and 1 file per region.

how big is the improvement in GC time, and in performance?

This patch has a lot of duplicated code.
And how do you guarantee that columns are sorted?

And how do you guarantee that columns are sorted?

a separate fn like toHBaseBulkWithPresortedColumns. any better ideas?

@Francesco:- Yeah i agree about the duplicated code in the toHBaseBulk(),
we could take common code out.
For column sorting issue :- I have written in the function javadocs that it
requires the columns to be sorted for now "Overloaded function, which
requires that sequence of headers and sequence of columns are in the sorted
order as required by HBase (i.e. sorted by columns names)....".

One way is what vidma suggested to rename the function,
Another option could be to use HBase KeyValue.COMPARATOR on the column
headers received and then sort the columns accordingly. Although this would
introduce another step.

Also, give me some time i will share the GC logs for both cases (with and
without improvements).
On Tue, Aug 11, 2015 at 10:32 PM, vidma<EMAIL_ADDRESS>wrote:

And how do you guarantee that columns are sorted?
a separate fn like toHBaseBulkWithPresortedColumns. any better ideas?
—
Reply to this email directly or view it on GitHub
https://github.com/unicredit/hbase-rdd/pull/18#issuecomment-129969016.

I just compared the GC logs for both the scenarios. I compared the GC logs of all single container for same time duration with & without changes.
Before the changes:

GC is invoked 692 times during the 40 mins duration of the job.
Total real GC time is 101.91 secs

After the changes:

GC is invoked 354 times during the 40 mins duration.
Total real GC time is 35.67 secs.

It is evident that less objects are being created in this case requiring less GC.

I made a new implementation of toHBaseBulk that is based on cells rather than rows. Now the sorting is all made by repartitionAndSortWithinPartitions and less intermediate memory objects are created.
For now it is in branch bulkwrite.
Can you test it and see if there are any improvements?

We released new version 0.6.0, with a new implementation of bulk write