windows: provide a mechanism for handling RawSockets

By analogy with #226 and the EventedFd, presumably an EventedSocket would be the way to go.
If this were very straightforward I'd submit a pull request.  But implementing Evented looks to be a lot more complicated on Windows and I don't really grok what's required.

This is certainly much more tricky I think as the event loop is going to basically always have to make some sort of assumption about what it can do once it receives a completion notification. For example it currently means that some previously-registered callback should be invoked to generate some notifications to be delivered to Handler::ready.
I'm not 100% sure how many use cases there are for custom handles on a completion port (beyond those that should be in mio by default of course), so we may want to wait for the space to play out for a bit before designing around this. It may end up just being one or two extension methods on an event loop, but for now it's certainly more involved enough than Unix that it's not easy to "just implement"

Maybe it'll be helpful to know about my use case.  I'm writing a Rust wrapper around the c-ares library.  The library tells you what file descriptors / sockets it cares about, and you tell it when something happens. Clearly a good fit for an event loop (though meanwhile I should be able to get by with select or WSAPoll).
Waiting to see whether anyone else cares, and why, and how the mio code develops, and so on is fine by me.

Hm interesting! Could you explain to me a bit how it's supposed to work in c-ares? For example does c-ares submit I/O to the socket for you, and then you let it know when it finished? (does c-ares have internal buffer management?).
e.g., if you were to write with c-ares using IOCP manually, what would it look like?

Well, I know more about c-ares than about IOCP.
c-ares goes something like this:

user calls ares_query()
while in this call c-ares opens a socket, and calls send() and recv() as required
on getting EAGAIN / EWOULDBLOCK, c-ares tells the user to watch the socket
user can be told what's interesting through a callback.  Or instead of using this callback the user can explicitly ask c-ares which sockets should be watched, with either ares_getsock() or ares_fds()
user is then expected to call either ares_process() or ares_process_fd() when it has news about socket readiness, at which point c-ares retries send() / recv() and we go around the loop again

I believe that the code was originally written with the expectation that the user would select() for socket readiness and has since evolved to support more efficient (Unix-style) event loops eg poll() / epoll() - which probably explains why there are two or three interface choices in each of the the last two steps.
I've never written any IOCP code, so I can't say much about that.  It may even be that it's not possible to integrate an IOCP event loop with requests made by plain old send() / recv()?!
My impression is that eg libuv does manage to achieve this, but I could be wrong. Here is an integration of the python wrapper of c-ares with the python wrapper of libuv, which is admittedly pretty indirect evidence of integration with IOCP - I may be just wrong in believing that libuv uses IOCP here.
I guess an alternative approach for mio would be to support an event loop backed by either WSAPoll() or select().  It's understood that these approaches don't scale to large numbers of sockets, but that's not always a problem.  Eg I'd expect it to be fine for c-ares users; c-ares shouldn't be talking to more than a handful of servers therefore won't need more than a handful of sockets.

Hm ok, so I think that basically as-is c-ares can't be used with IOCP. This would require c-ares to actually execute the I/O and then ask you to tell it when the I/O completes, rather than the other way around (requesting you tell it when I/O can be completed). I believe that Python wrapper is just falling back to polling (e.g. the functions you found), so I don't believe it's using IOCP. That being said I'm not entirely sure how libuv would merge WSAPool and IOCP event loops...
I think something like this on Windows could certainly be supported for only sockets via the normal select and WSAPoll APIs, but it'd need some thought to figure out how to merge these event loops (perhaps it's just a worker thread)

There is a list of I/O operations that complete with IOCP.
When communicating with (custom) device driver, ReadFile, WriteFile and DeviceIoControl are used a lot. WaitCommEvent can also be used if the device is serial port.

I think that it would be valuable to consider this case when considering a more windows specific API but this will be post 1.0. I will flag it as such.

Windows is actually in a relatively good position for handling this at this point. With the Registration and SetReadiness types it's easy to integrate I/O readiness from any object into the mio event loop.
The completion status that we get from IOCP gives the event loop two ways to configure it, the token and the overlapped pointer. Unlike Unix we don't know how to translate that to read/write readiness events, so we have to run some form of callback to actually handle the event that finished to figure out what to do.
Right now we do a "clever trick" where the overlapped structure used internally ubiquitously contains a function pointer just after the OVERLAPPED structure. This means that the pointer we get from IOCP we just cast to one of those instances, and then we call the function pointer on the completion status we just got. This is all pretty unsafe, but also a very small API commitment and flexible enough to implement TCP/UDP at least. I could imagine something like:
pub struct EventedHandle<'a> {
    inner: &'a HANDLE,
}

impl<'a> EventedHandle<'a> {
    // Unsafe because IOCP is super unsafe in general.
    pub unsafe fn new(handle: &'a HANDLE) -> EventedHandle<'a> {
        // ...
    }
}

impl<'a> Evented for EventedHandle<'a> {
    // ...
}

Then after that all I/O would just have to be done with the custom overlapped structure (hence the unsafety of the constructor). We may have to expose one or two other fiddly bits, but overall it should be pretty similar to Unix I think!

I've got a sample implementation of this (different than what I last mentioned) at https://github.com/carllerche/mio/pull/504, comments would be welcome!

I've only skimmed over that code - there's quite a lot of it and I suspect it would take me a while to get to grips with it!  But I don't immediately see how it deals with the particular use case that I had in mind when I raised this issue.
Reminder:

c-ares calls socket(), and instructs me "I'd like you to tell me about events on this file descriptor / raw socket".
I'd like to be able to fulfil that instruction via mio.

That's easy on unix because file descriptors are Evented, but not currently possible on windows.
Does #504 make this possible?  Is it easy?!

@dimbleby ah so for integration with c-ares, which I imagine is very similar if not the same as libcurl (by the same author), is impossible to do with mio. Fundamentally mio uses IOCP, whereas libcurl (and I assume c-ares) desires notifications of readiness. The mio library will perform I/O on your behalf, whereas libcurl expects to do all I/O itself.
You can see an example of working across Unix/Windows with libcurl at https://github.com/tokio-rs/tokio-curl. It basically boils down to a worker thread for Windows whereas Unix can integrate with the main event loop.
So in that sense, if you want readiness of a socket on Windows without performing I/O, I think that mio is very unlikely to ever support.

Looks like this wasn't fixed in #504

@alexcrichton thanks, I have much the same windows / not-windows split in https://github.com/dimbleby/c-ares-resolver (which is basically the c-ares wrapper integrated with an event loop).
As we discussed much earlier in this thread, it would be possible for mio to fall back to a worker thread for just this case.  While that would be kinda desirable I think, I'm neither planning on contributing this myself nor holding my breath on anyone else doing so.  If and when mio maintainers decide this is never going to happen, I expect you should just close this one out.

@dimbleby ah ok, if the worker thread approach is the one you'd think of taking, I suspect that probably won't make its way to mio itself. Something like that can definitely be built outside of the mio crate itself (e.g. the tokio-curl crate I linked) and should be easily integrate-able with the Readiness and SetReadiness structures in mio today.

(in that case I'm going to close)


11 joined people are shown in the Events message

Only 10 (or maybe even less like 5) should be shown and if more join, add a ...

Fixed with 8bb2b12347ee9b4646bcb684be2823ee92a8075d


:sparkles: Add independently verifiable contentHash to Essay NFT

essayUrl becomes contentHash and metadataUri — the independently verifiability of the sha256 content hash is powerful. otherwise users/collectors won't be able to verify content hasn't changed unless they understand IPFS content addressing rules.

✨ ✨ ✨


victron Operation failed with ATT error: 0x0e (Unlikely Error)

5:45:56 INFO [sampling] Bleak version 0.20.2
15:45:56 ERROR [main] Error (num 6, max 200) reading BMS:
15:45:56 ERROR [main] Stack: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/bleak/backends/bluezdbus/client.py", line 204, in connect
    reply = await self._bus.call(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/dbus_fast/aio/message_bus.py", line 390, in call
    await future
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/app/main.py", line 39, in fetch_loop
    await fn()
  File "/app/bmslib/sampling.py", line 90, in __call__
    return await self.sample()
           ^^^^^^^^^^^^^^^^^^^
  File "/app/bmslib/sampling.py", line 112, in sample
    async with bms:
  File "/app/bmslib/bt.py", line 325, in __aenter__
    await self.connect()
  File "/app/bmslib/models/victron.py", line 80, in connect
    await super().connect(timeout=timeout)
  File "/app/bmslib/bt.py", line 221, in connect
    await self._connect_client(timeout=timeout)
  File "/app/bmslib/bt.py", line 176, in _connect_client
    await asyncio.wait_for(self.client.connect(timeout=timeout), timeout=timeout + 1)
  File "/usr/lib/python3.11/asyncio/tasks.py", line 489, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/bleak/__init__.py", line 531, in connect
    return await self._backend.connect(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/bleak/backends/bluezdbus/client.py", line 141, in connect
    async with async_timeout(timeout):
  File "/usr/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError
15:45:56 INFO [sampling] connecting bms SmartShuntBt(E0:E5:16:A0:5A:C8,inverter_in)
15:46:00 ERROR [base_events] Task exception was never retrieved
future: <Task finished name='Task-5161' coro=<SmartShuntBt._keep_alive_loop() done, defined at /app/bmslib/models/victron.py:58> exception=BleakDBusError('org.bluez.Error.Failed', 'Operation failed with ATT error: 0x0e')>
Traceback (most recent call last):
  File "/app/bmslib/models/victron.py", line 63, in _keep_alive_loop
    await self.client.write_gatt_char('6597ffff-4bda-4c1e-af4b-551c4cf74769', data, response=False)
  File "/usr/lib/python3.11/site-packages/bleak/__init__.py", line 659, in write_gatt_char
    await self._backend.write_gatt_char(char_specifier, data, response)
  File "/usr/lib/python3.11/site-packages/bleak/backends/bluezdbus/client.py", line 905, in write_gatt_char
    assert_reply(reply)
  File "/usr/lib/python3.11/site-packages/bleak/backends/bluezdbus/utils.py", line 22, in assert_reply
    raise BleakDBusError(reply.error_name, reply.body)
bleak.exc.BleakDBusError: [org.bluez.Error.Failed] Operation failed with ATT error: 0x0e (Unlikely Error)

This error sometimes appears but doesn’t stay for a long time.
RPI 4, bleak 0.20


Schema for remaining EPAC objects

Create schemas for:

global-settings.jsonc
Policy definition (only top-level)
Policy Set definition (complete)
Policy Exemption

Fixed minor issues with the schemas


Have belay update print each dependency on a single line

Codecov Report

Merging #104 (667312a) into main (3ec4fbf) will decrease coverage by 0.14%.
The diff coverage is 50.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main     #104      +/-   ##
==========================================
- Coverage   77.45%   77.31%   -0.14%
==========================================
  Files          31       31
  Lines        1224     1221       -3
  Branches      210      210
==========================================
- Hits          948      944       -4
- Misses        210      211       +1
  Partials       66       66

Flag
Coverage Δ

unittests
77.31% <50.00%> (-0.14%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

belay/packagemanager/group.py
75.28% <50.00%> (ø)

belay/cli/cache.py
30.43% <0.00%> (-4.35%)
:arrow_down:

belay/executers.py
82.65% <0.00%> (-0.18%)
:arrow_down:

belay/hash.py
100.00% <0.00%> (ø)

belay/cli/identify.py
28.00% <0.00%> (+1.07%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


commit swagger generated files

Codecov Report
Patch and project coverage have no change.

Comparison is base (fea4ce1) 42.45% compared to head (44e81dc) 42.45%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #282   +/-   ##
=======================================
  Coverage   42.45%   42.45%
=======================================
  Files          14       14
  Lines         954      954
=======================================
  Hits          405      405
  Misses        512      512
  Partials       37       37

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


variables in php double quote is not highlighted

VSCode Version: Code 1.8.1 (ee428b0eead68bf0fb99ab5fdc4439be227b6281, 2016-12-19T14:42:49.304Z)
OS Version: Darwin x64 16.3.0
Extensions:

Extension
Author
Version

vscode-theme-onedark
akamud
0.2.14

vscode-great-icons
emmanuelbeziat
1.1.37

ftp-simple
humy2833
0.4.14

vscode-icons
robertohuertasm
6.0.0

Steps to Reproduce:

type "hello, $user" in php file
see the problem

I think the php content has to be inside of a <?PHP block:

After adding that, I see the correct colorization


Behavior on retry

How do you envision the retry working in detail?  Would the browser retry a POST request?  Would it honor Set-Cookie headers sent in the first response?

we only modify the accept-language header with the more preferred language for the retry request. For other headers it would be the same as the initial requests. I don't think we change the request method, it depends on the initial request as well.  You can see more details on the implementation docs: https://docs.google.com/document/d/1RkPDf7DNtcOj4KXeW8wNCuYfto-drnGYST_NvZe3GoY/edit?tab=t.0#bookmark=id.jbb4wdao0qq9


Feature Request: visual mode

Thanks for this plugin.  Super useful.
Do you think it might be good to have a visual mode version?  That would allow us to set up a logging statement for an expression instead of for a particular variable.  Again, awesome job!

Hi,
There is now an experimental support for this.
It will print the content of your selection, using the first node of the selection to choose where to place the console.log or print statement.


Hue motion sensor SML001 improved

This is a fix for Hue motion sensor SML001.
The previous implementation has a serious problem: It fails if there is continuous motion.
Since there was a commit addressing an issue before already, here a summery (solution below):
The original implementation before  this commit ignored the reported value and published "motion true" on any event, even if the sensor report said "motion false".
Resulting in:

Time X: Single Motion, Sensor sends "occupancy: 1" -> publish "Motion true", occupancy_timeout to 90 seconds	(CORRECT)
X+10sec: Sensor sends "occupancy: 0" -> converter IGNORED value "0", published "Motion true" (WRONG), occupancy_timeout got reset to 90 seconds
X+10+90: occupancy_timeout occures, publish "Motion false" by converter	(more or less CORRECT)
every 600sec: Sensors reports current status, "occupancy 0", value "0" IGNORED again, converter published "Motion true" 	(WRONG)
(Thats the 600 in devices.js "(cb) => device.report('msOccupancySensing', 'occupancy', 0, 600, null, cb),")

After the mentioned commit:

...
X+10sec: Sensor sends "occupancy 0" -> converter ignores all except value "1" -> nothing published or reset (WRONG)
X+90sec: occupancy_timeout occures, publish "Motion false" by converter
every 600sec: Sensor reports "occupancy 0", ignored because not "1", nothing published, issue seems solved, because no wrong events even if sensor stored in box...

Now the current issue:

...
X+x+xsec: continous motion, sensor resets every time its internal timer. No attrReports are send because state does not change (still occupancy 1/true)
X+90sec: occupancy_timeout in converter occures -> publish "Motion false" 	(WRONG, sensor is still on occupancy true)
The timeout gets only reset if there is no motion for > 10 sec, because Sensor changes to "0" (report ignored by converter) and at next motion to "1" (report accepted, timeout reset)

Solution:
No occupancy_timeout timer in converter! We just handle the reported value.
The hue sensor sends both messages, occupancy true (motion detected) and false after no motion was detected for 10 seconds (default setting). So we don't need the occupancy_timeout timer.
I made lots of tests, the motion start/stop messages are correctly reported by the sensor. And if the stop message may be missed once (lost packet), the next periodic report (the 600 sec) will publish correct value. (I changed this value to 90% of occupancy_timeout, just in case...)
Here a log of the occupancy related events the sensor is generating (pirOToUDelay: 0, reporting max intervall: 600)

2018-12-31T07:11:58.423Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 1 } ]
2018-12-31T07:11:58.424Z   ind:changed                                  { cid: 'msOccupancySensing', data: { occupancy: 1 } }
2018-12-31T07:12:00.043Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 1 } ]
2018-12-31T07:12:08.424Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T07:12:08.424Z   ind:changed                                  { cid: 'msOccupancySensing', data: { occupancy: 0 } }
2018-12-31T07:12:10.019Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T07:22:08.174Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T07:22:09.771Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T07:32:08.086Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T07:32:09.689Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T07:42:07.967Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T07:42:09.565Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T07:52:07.877Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T07:52:09.477Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T08:02:07.695Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T08:02:09.297Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T08:12:07.388Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T08:12:08.985Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
...
2018-12-31T08:31:23.790Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T08:32:46.571Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 1 } ]
2018-12-31T08:32:46.572Z   ind:changed                                  { cid: 'msOccupancySensing', data: { occupancy: 1 } }
2018-12-31T08:32:48.184Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 1 } ]
2018-12-31T08:36:06.075Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]
2018-12-31T08:36:06.077Z   ind:changed                                  { cid: 'msOccupancySensing', data: { occupancy: 0 } }
2018-12-31T08:36:07.701Z   ind:reported   msOccupancySensing            [ { attrId: 0, dataType: 24, attrData: 0 } ]

motion start is reported
after 10 seconds motion stop is reported
every 600 seconds a current status report (twice)
the last event was 3 minutes permanent motion in front of sensor

(only ind:reported is used by converter)
Note: An "occupancy timeout" can be configured by setting sensors pirOToUDelay value. I worked on this already, will be submited in second pull request.
I moved the original converter implementation to generic_occupancy_no_off_msg, so behavior for other sensors will not change.

:+1:
I just wanted to create an issue for this since I'm also observing this behavior and it seems incorrect to me.
Just did a bunch of tests and my observations match yours:
Trigger motion once:
t=0 attReport + devChange occupancy 1
t+2 another attReport occupancy 1 (not sure why this one happens)
t+10 attReport + devChange occupancy 0
t+12 another attReport occupancy 0 (again not sure why this one happens)

Trigger motion continuously for 60s:
t=0 attReport + devChange occupancy 1
t+2 another attReport occupancy 1 (not sure why this one happens)
t+70 (i.e. 60s motion + 10s delay from the sensor itself) attReport + devChange occupancy 0
t+72 another attReport occupancy 0 (again not sure why this one happens)

So what the sensor does is the following:
- Detect motion
- Send occupancy 1
- (If there's continuous motion) wait until there's no motion detected
- Wait 10s
- Send occupancy 0

This behavior from the device combined with the behavior from zigbee2mqtt means that it's currently impossible to handle continuous motion (which means the lights turn off even when I'm still in the room).
I like the proposed solution, no abstractions, just forward the value as reported by the sensor. Very much prefer this over having logic around this in zigbee2mqtt which I see as a simple conversion that shouldn't contain any logic. This way everyone can add their desired logic around this themselves in their chosen solution.
I do believe @Koenkk mentioned the occupancy_false reporting always happening was by design, so maybe he has a different opinion on it.

occupancy false should only be generated for motion sensors that do not send it, meaning this implementation is good.
I've the comment above is fixed this can be merged.


Reduce unnecessary fiber usage

This should at least reduce memory usage and fiber spam.

Maybe this is the wrong level to optimize at, and we should rather optimize in LocalRest / Vibe.d. Although that means optimizing in two different codebases..

This seems to mostly work minus the CircleCI troubles. I did get some failures previously in the Github CI, but I don't see where the problem is - seems random.

It's green now.


Add Cell Render

?

Cell Render = Even if u are at What Ever Mass
Like Classy :)
@Megabyte918 w0w ur always active :3 <3

#217 Client sided issue, this has nothing to do with the server.

@NameYT It is client sided, just zoom out 50% on the vanilla client (by zoom i mean ctrl + minus) and it will look the same, however @AlexandriteErikParacelsus is right you do need a pretty good computer for it.


Create a SAP Fiori App and Deploy it to SAP S/4HANA Cloud, ABAP Environment: Create IAM App and business catalog

Tutorials: https://developers.sap.com/tutorials/abap-s4hanacloud-procurement-purchasereq-shop-ui.html
In step 10 package is ZTRAVEL_APP_XXX but should be Z_PURCHASE_REQ_XXX. Please adjust accordingly.

Hi @peterpersiel ,
thank you for your feedback. I created following PR: #20484 for this GitHub issue.
Best regards,
Merve


Simplify Cert-Manager Installation using the official one-line YAML installation method

This issue is a:

[ ] Bug Report
[x] Enhancement suggestion
[ ] Feature request

The current cert-manager installation is outdated, and currently doesn't work as intended in the current K3S version. A companion PR,  GH-160  is created to use the official one-line YAML installation instead of the usual Jetstack Helm chart.

Ref guide https://cert-manager.io/docs/installation/kubernetes/#installing-with-regular-manifests

Already done via GH-160. Closing.


Do Not Merge: Deploy

This is an automatically generated release PR!

Holding off on this deploy temporarily while we investigate an issue in staging.


OpenSslEngine with no finalizer

Motivation:
OpenSslEngine and OpenSslContext currently rely on finalizers to ensure that native resources are cleaned up. Finalizers require the GC to do extra work, and this extra work can be avoided if the user instead takes responsibility of releasing the native resources.
Modifications:

Make a base class for OpenSslENgine and OpenSslContext which does not have a finalizer but instead implements ReferenceCounted. If this engine is inserted into the pipeline it will be released by the SslHandler
Add a new SslProvider which can be used to enable this new feature

Result:
Users can opt-in to a finalizer free OpenSslEngine and OpenSslContext.
Fixes https://github.com/netty/netty/issues/4958

@rkapsi @normanmaurer @trustin  - PTAL

This is the implementation of https://github.com/netty/netty/pull/4963#issuecomment-197731489

@Scottmitch LGTM - it's big but straight forward and mostly just moving things around. I'm not too concerned about finailzers as long as I can preemptively free resources myself (like close() on Sockets, FileInputStreams etc).
I'm happy to share with you via email why reference counting is important for us. It'd make a great Netty blog post or something but I assume you can deduct it from the tea leafs. ;)

it's big but straight forward and mostly just moving things around

+1. Only a few "changes" as a result of CI warnings.

I'm happy to share with you via email why reference counting is important for us. It'd make a great Netty blog post or something but I assume you can deduct it from the tea leafs. ;)

I'm always interested in hearing more about Netty use cases ... the more informed we are the better we can support Netty users.  Feel free to reach out if you are willing to share :+1:

+1

Am 24.07.2016 um 15:15 schrieb Scott Mitchell<EMAIL_ADDRESS>it's big but straight forward and mostly just moving things around
+1. Only a few "changes" as a result of CI warnings.
I'm happy to share with you via email why reference counting is important for us. It'd make a great Netty blog post or something but I assume you can deduct it from the tea leafs. ;)
I'm always interested in hearing more about Netty use cases ... the more informed we are the better we can support Netty users. Feel free to reach out if you are willing to share 👍
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@buchgr you were talking about this?
while ((object=findUnreachableObject())!=null) {
    if (object.metadata.is_finalized==NEEDS_FINALIZE) {
        if (hasNonNullBody(object.finalize)) {
            Finalizer.addForProcessing(object);
            object.metadata.is_finalized=IN_FINALIZER_QUEUE;
        } else {
            object.metadata.is_finalized=REMOVE_NOW;
        }
    }
    if (object.metadata.is_finalized==REMOVE_NOW) {
        // destroy the object and free the memory
    }
}

@normanmaurer - wdyt?

Left some comments, mostly minor.

@trustin @normanmaurer - Comments resolved ... ok to merge?

@Scottmitch just a few questions then LGTM

4.1 (7d77458) 4.0 (c1c17dd)


Add a REST API to control onionize via browser or curl

And eventually drop GTK3 support.

see also #63


new feature idea: auto-generation of angular services and html forms based on tastypie-swagger or any json schema

Hi,
First, kudos for this lib, it's a huge time saver!
I wonder if - w/ or w/o a json schema - we could enhance this lib to "auto-discover" tastypie models, and thus, auto-generate them as angular.js services.
Since tastypie-swagger is basically guessing the schema for documentation purposes, one might hijack that to auto-generate angular services based on your lib, avoiding to write boilerplate code at the js layer (basically declaration of angular services, and most of the CRUD forms' html/angular directives code).
What do you think?

Hi.
Thank you.
I find this very good idea. I had already thought of that, but when I met the "angular material" that contains many ready directives, left it aside. Today it is so easy to use frameworks (bootstrap, angular material, Younic), that my form would hinder rather than help. lol. And also for quick entries, there is a "django admin" that works very well with an administrative support.

Hi Marco,
In fact, I think it's complementary to django admin, which is suitable for backend admin (not using tastypie/drf at all), whereas the kind of view that you did using material, is the one that people tend to give to the end user, with more restriction applied. Right now, when one wants to consume a rest api from a SPA app made w/ angular, he has to write lots of boilerplate code. This is where a generic version of your lib+angular material sample app could give a out-of-the-box admin UI for the end user, as opposed to django admin which is too dangerous to give to the end-user. It has the advantage to follow the eat-your-own-dog-food approach regarding API (kind of virtuous cycle).
If i've time in the coming days, i'll look into that, because it's a repeating pattern I'm having and probably not the only one ;-)

So the context of my lib is generic for developers, not for end User. The programmer can consume web services and connect to your frontend any. There are endless possibilities for frontend. My lib is only the model of MVC, has no directives (view). For example, you said that this repeating patterns, what framework you're using for the layout? This layout is your project, the lib have to keep generic for any project (layout). I think we have to keep it that simple. I like your idea, but for a new project on top just this context, creating an admin for tastypie using my lib. What do you think?

yeah, you're absolutely right, better to do that outside of the scope of this lib.
and it's really a material design admin UI for tastypie that i've in mind, probably using built-in schema of tastypie

Ok, thanks for the use of the lib. I'm having the same problem as you, lack of time to start new projects. We can start this project together soon, I liked the idea.
See you.


Fix #712, allow overwriting the public IP of a Kubernetes node

Description
This PR fixes #712 by allowing to overwrite the public ip of  a Kubernetes node via the flannel.alpha.coreos.com/public-ip-overwrite annotation.
Can anyone point me to the appropriate place for docs, is it just Documentation/Configuration.md?
Right now the other annotations aren't documented in this repo.
Todos

[ ] Tests
[ ] Documentation

@alvaroaleman Thanks for this change. If you can add some documentation and ideally a test, I can get it merged. For the documentation, I would suggest adding an "Annotations" section to https://github.com/coreos/flannel/blob/master/Documentation/kubernetes.md

Added docs. Writing a test seems to be rather complicated, because there are none for the kube subnet manager yet.

I added some tests for kube subnet manager recently - https://github.com/coreos/flannel/blob/master/dist/functional-test-k8s.sh
But I agree, they are still quite complicated!

Thanks for the hint @tomdee, I added a test.
I also moved the check for ls /run/flannel/subnet.env into the start_flannel function, since it is a generic test if flannel startup was finished and not specific to the other ping tests in the suite.

Recreated the PR because travis was failing after having issues pulling the etcd image.
The e2e tests run twice btw, because they are called in the travis file and from within the test maketarget

OK, this looks good. It would be great if you could add a little more information to the documentation, such as why a user might want to set this, and any other annotations that can be set.
Before I merge can you squash your commits so I'm just merging a single commit?

Updated the explanation and squashed my commits.
As far as I can tell, the other annotations are not supposed to be altered by users thus it doesn't make sense to document them or am I wrong there?

Yes, I think you're right! The tests are now failing, I think you lost a commit when doing the squash? Hopefully you can get it back easily then I can merge.

Added the missing commit back.


Remove rollbar module

Rollbar module is currently broken when used with recent versions of the Rollbar gem. Monkeypatching the Rollbar seems like a bad practice as we can configure custom fingerprinting on Rollbar directly now. More details on #122 .
This PR:

Removes rollbar module
Adds instructions to configure Custom Fingerprinting directly on Rollbar.

Another approach would be to deprecate the module first, and then remove it. As this feature was optional and experimental, removing it seems better. Thoughts?

It's break the require for anyone who was using it, which will cause the app to fail to boot. Could leave the file in there and have it echo an error message instead, but I'm not particularly bothered either way. As you said, it was an experimental feature. Not sure how much use it has in practice.

Added a warn message stating that the module was removed.

That works for me! We can remove the file in 0.6.0, if we ever get there.
Thanks so much for the contribution! :100:

This should perhaps be mentioned in https://github.com/heroku/rack-timeout/blob/master/UPGRADING.md

Good point, I'll add something!


Store pending messages on broker's side

Describe the feature request
I need to store messages pending in server's queues, that are not yet delivered to subscribers. And when server crashes, I would like to restore these queues to be sure that every message reaches each subscriber.
Which project is your feature request related to?

Server

Describe the solution you'd like
I need to store messages pending in server's queues, that are not yet delivered to subscribers. And when server crashes, I would like to restore these queues to be sure that every message reaches each subscriber.
For example, when client (subscriber) is disconnected for some time, its queue on server side is populated with new pending messages. And when server crashes at this point, subscriber will lose those messages. Even when the server restarts and subscriber's reconnect will take place, messages are lost.
Additional context
Place of storing should be decided on my own. It may be file, database or whatever.
@saba-sabrin I think we just store the messages in memory, so if the broker is restarted you will still loose messages.
@chkr1011 should we extend IMqttServerStorage or provide a similar interface like IMqttSessionStorage   to store pending messages of a session?
Originally posted by @JanEggers in https://github.com/chkr1011/MQTTnet/issues/498#issuecomment-448552957

This feature is still in progress but unfortunately it is hard to implement. Especially when performance should by high and feature set rich. The problem is when should the messages being stored? Every time a message is produced? Then the broker will get slow when a lot of traffic is happening. But when storing the messages on a clean exit of the server you will lose all messages on an app crash etc.

I wanted to comment this is an important feature.  I understand the performance implications.  Other brokers offer session persistence on stop/start scenarios...I'm not sure how they work on crash protection.  Since there will be a performance impact for a crash protection setup, then it could be designed as an optional feature to turn on/off so it could be implemented by the user depending on their use case.  Right now I'm using mqtt for a platform that won't have extremely high throughput, but session persistence is very important.
Thanks for your work on this project.

There are quite a few touch points in code where the "Persistent Session Manager" interface needs to be called but it all looks good so far. Because I have been working on the interface only (not the storage behind the interface), I think I'll open another issue or pull request "at the right time". I suspect that a good time for a pull request will be when the Reason_Code_In_Subscription_Interceptor branch is merged into master.

@logicaloud This is a feature that I would very much like to utilize. What is the current status?

@mark-trumed The MqttPersistedSessionManager is currently implemented in fork https://github.com/logicaloud/MQTTnet/tree/persisted-sessions-v4 but somewhat tangled up with some other added MQTT5 features like session and message expiry and improvements for retained message handling. My thinking was to untangle the features then create pull requests separately, closer to the end of the year when I have more time. I'm not sure how this would fit in with current ongoing developments. @chkr1011, any comments are welcome.

@logicaloud @chkr1011 Persistence is mandatory for any application where data integrity is nonnegotiable (no matter the performance impact). Is there any update on the progress?
Since I need this feature I have to try to implement a crude solution myself.
My plan was to persist both the subscriptions and the unfinished messages. Do I also need to persist the Sessions? And if so, how can I get the sessions and re-inject them on startup?
Furthermore, how do I know if a message was delivered to all clients and is dequeued by the server? My understanding is that the server stores all messages (with QoS > 0) in some collection and removes them from the collection once all subscribed clients received the message. Is there an event for that? I don't think "ClientAcknowledgedPublishPacketAsync" is what I'm looking for, however I could be wrong since there is no documentation.

I'm a bit in limbo with this at the moment and I think the ball is in @chkr1011's court. There is a discussion going on over here: https://github.com/dotnet/MQTTnet/discussions/1764.
As mentioned there, the persistent messages and persistent sessions integration should probably use interfaces instead of events. Once that is agreed on, then I would like to update the branch referred to in the discussion to change from an event based approach to using interfaces, then tidy up, review, and create some pull requests. The end result would provide suitable extension points within the MQTTnet library to hook up storage providers like Redis or SQL databases (to be implemented in a separate library). The integrated default implementation would use in-memory retained messages and persistent sessions, so the built-in features would be limited but directly usable.
I'm not sure about whether there is a notification about all messages being delivered; if not then that maybe that is something that could be considered going forward.


perf: kmean fit uses cached torch dataset

Trained on IVF65535,PQ192 on a Titan RTX,  finished IVF part training in 2390s.


Add 'ia_content' user to DevX owned gems

Ensure that the shared IAC Rubygems account is an owner on the following repos:

[x] pdk
[x] puppet-strings
[x] puppet-resource_api
[x] puppetlabs_spec_helper
[x] dependency_checker
[x] pdksync
[x] rspec-puppet
[x] puppet-lint
[x] ruby-pwsh

We managed to get access to the dependency_checker gem


rio 0.2.0

Checklist for Pull Requests

[x] Have you signed the Contributor License Agreement?
[ ] Is there a linked Issue?

Manifests

[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.6 schema?

Note: <path> is the directory's name containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Automatic Validation ended with:

Installation failed with exit code -1978334957
2024-11-13 14:28:13.914 [CLI ] ShellExecute installer failed: 1633
MSI (s) (84:94) [14:28:05:033]: PROPERTY CHANGE: Adding OutOfDiskSpace property. Its value is '0'.
MSI (s) (84:94) [14:28:05:033]: PROPERTY CHANGE: Adding OutOfNoRbDiskSpace property. Its value is '0'.
MSI (s) (84:94) [14:28:05:033]: PROPERTY CHANGE: Adding PrimaryVolumeSpaceAvailable property. Its value is '0'.
MSI (s) (84:94) [14:28:05:033]: PROPERTY CHANGE: Adding PrimaryVolumeSpaceRequired property. Its value is '0'.
MSI (s) (84:94) [14:28:05:033]: PROPERTY CHANGE: Adding PrimaryVolumeSpaceRemaining property. Its value is '0'.
Property(S): OutOfDiskSpace = 0
Property(S): OutOfNoRbDiskSpace = 0
Property(S): PrimaryVolumeSpaceAvailable = 0
Property(S): PrimaryVolumeSpaceRequired = 0
Property(S): PrimaryVolumeSpaceRemaining = 0
MSI (s) (84:8C) [14:28:05:830]: PROPERTY CHANGE: Adding OutOfDiskSpace property. Its value is '0'.
MSI (s) (84:8C) [14:28:05:830]: PROPERTY CHANGE: Adding OutOfNoRbDiskSpace property. Its value is '0'.
MSI (s) (84:8C) [14:28:05:830]: PROPERTY CHANGE: Adding PrimaryVolumeSpaceAvailable property. Its value is '0'.
MSI (s) (84:8C) [14:28:05:830]: PROPERTY CHANGE: Adding PrimaryVolumeSpaceRequired property. Its value is '0'.
MSI (s) (84:8C) [14:28:05:830]: PROPERTY CHANGE: Adding PrimaryVolumeSpaceRemaining property. Its value is '0'.
MSI (s) (84:8C) [14:28:13:783]: Product: Rio -- Installation failed.
This installation package is not supported by this processor type. Contact your product vendor.
Installation failed with exit code -1978334972
2024-11-13 15:13:36.073 [CLI ] No suitable installer found for manifest Microsoft.VCRedist.2015+.x64 with version 14.42.34433.0

Automated error analysis suggests -1978334957 may mean Package not supported by the system.
Automated error analysis suggests -1978334972 may mean This package has a dependency missing from your system.
Automated error analysis suggests 1633 may mean ERROR_INSTALL_PLATFORM_UNSUPPORTED - This installation package is not supported by this processor type. Does this manifest have Architecture: x86? It might need to be x64.
(Automated response - build 897.)


Background gradient

Taking the animated gradient background from #47
Making the modal card slightly smaller and position it closer to the top

We also need to change the text color in about page to make it white or lighter to keep the contrast.


App tables in full build not working correctly.

Seems to be related to requiring a cell template

Seems like ui-grid cannot use a cell template defined via a require or an import.  It must be inlined.  Not sure if this is a webpack thing or ui-grid thing.


UnboundLocalError: local variable 'HTML_OUTPUT_FILE' referenced before assignment

Traceback (most recent call last):
  File "/Users/jwr003/repos/gqa/venv/bin/tuih", line 8, in <module>
    sys.exit(main())
  File "/Users/jwr003/repos/gqa/venv/lib/python3.9/site-packages/pytest_tui/html_gen.py", line 360, in main
    with open(HTML_OUTPUT_FILE, "w+") as f:
UnboundLocalError: local variable 'HTML_OUTPUT_FILE' referenced before assignment```

Unable to reproduce, but added safeguard code (making HTML_REPORT_FILE global) to make sure. Commit 707853a..0e7f5d4.


Fails to download the Server ISO

Please see the error message.

I have verified that I'm on the Win10 Enterprise version. This is a physical machine, and I have adjusted the power setting to high performance. I have watched the download happen 3 times, and seems to not get all the bits. when I go back and check to see the ISO's you can see that not all the bits are there.

please advise as to a possible solution.
Thanks,
Jay

I'm going to try and run the setup-lab instead of the unattend.lab and see if that makes any difference.
Also, I do some digging on the error, and on line 357 char 27 all looks good, but maybe someone can take a look.

I have the same problem. I was able to download it a couple of days ago, but deleted the environment due to other issues so I no longer have it.

When can we expect to see an answer to this issue, or at least an acknowledgement that this is an issue.

This project is now under new ownership. The 3.x version line is obsolete. Version 4.0.0 has been published to the PowerShell Gallery with new updates, configuration, and documentation. It is expected that this version will resolve most if not all of the problems you were experiencing. Read Updating for more information.
No further action will be taken on this issue and it will most likely be closed at some point. If you still have the same issue with the new version, then go ahead and open a new issue.

For future reference, this configuration has been marked as obsolete, although we may need to build a replacement. Follow the instructions to remove any old versions of the module, install the new version from the PowerShell Gallery and try again. If you still run into problems, then please post a new issue.


#1644 Add button for VirtualizedTable Rows in Expression Builder

Fixes: #1644
Developer's Certificate of Origin 1.1
   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

@matthoward366

Add column width: Could you please check now, I have made the width to be in px, so that it's used as is and not adjusted.
Column Sort: Unable to replicate the issue.

Here is what I'm seeing for sort.  The selection gets off.


When is the TypeInterceptor used?

Curious when exactly it's being used, as my code never utilizes it.

Hi, the type interceptor is used when the OnBeforeCompleteType is invoked.
In order for your code to utilize this, you will need to ensure your models are annotated and the AddDataAnnotationsValidator method is added after the AddGraphQLServer.
Hope this helps!

That's the thing - in my local environment I've completely removed the type interceptor and everything still works. There is no dictionary in the context data with the name "ValidationAttributes" though, but in my case it was never even used?

If memory serves correct, HotChocolate utilizes the type TypeInterceptor to decorate fields with behaviors.
In the DataAnnotatedModelValidations case, either to exclude something from validation or to add any ValidationAttributes.


Highlighting not working for SAS language option

Hi Anas!
Thanks for a great project, it's been super useful!
I am displaying SAS code, and noticed that the highlighting is not working. The code is white, as if "text" was chosen as the programming language. I am also displaying sql and yaml code in different editors on the same page, which are both working nicely.
Code for SAS editor:
sas_dict = code_editor(
                    ss["job"].sas_code,
                    shortcuts=ss["shortcuts"],
                    theme=ss["theme"],
                    lang="sas",
                    height=ss["height"],
                    buttons=code_editor_buttons,
                    options={"wrap": ss["wrap"], "showLineNumbers": True},
                    info=info_bar_sas,
                )

where
# Code editor settings
with st.expander(
        "Code editor settings" if english() else "Instillinger for kode-editor",
        expanded=False,
    ):
        col_a, col_c, col_cb = st.columns([17, 3, 3])
        col_c.markdown(
            '

', unsafe_allow_html=True
        )
        col_cb.markdown(
            '

', unsafe_allow_html=True
        )

        ss["height"] = col_a.slider("min-max lines:", 1, 100, (40, 45))

        col_d, col_e, col_f = st.columns([1, 1, 1])
        ss["theme"] = col_d.selectbox(
            "Theme:", ["default", "light", "dark", "contrast"], index=2
        )
        ss["shortcuts"] = col_e.selectbox(
            "Shortcuts:", ["emacs", "vim", "vscode", "sublime"], index=2
        )
        ss["focus"] = col_c.checkbox("Focus", False)
        ss["wrap"] = col_cb.checkbox("wrap", False)
        )

code_editor_buttons = [
    {
      "name": "Copy",
      "feather": "Copy",
      "hasText": true,
      "alwaysOn": true,
      "commands": [
        "copyAll",
        [
          "infoMessage",
          {
            "text": "Copied to clipboard!",
            "timeout": 2500,
            "classToggle": "show"
          }
        ]
      ],
      "style": {
        "top": "-0.25rem",
        "right": "5.4rem"
      }
    },
    {
      "name": "Save",
      "feather": "Save",
      "hasText": true,
      "bindKey": { "win": "Ctrl-S", "mac": "Command-S" },

      "alwaysOn": true,
      "commands": [
        "save-state",
        [
          "response",
          "saved"
        ]
      ],
      "response": "saved",
      "style": {
        "top": "-0.25rem",
        "right": "0.4rem"
      }
    }
  ]

info_bar_sas = {
        "name": "language info",
        "css": css_string,
        "style": {
            "order": "1",
            "display": "flex",
            "flexDirection": "row",
            "alignItems": "center",
            "width": "100%",
            "height": "2.5rem",
            "padding": "0rem 0.75rem",
            "borderRadius": "8px 8px 0px 0px",
            "zIndex": "9993",
        },
        "info": [{"name": "SAS", "style": {"width": "100px"}}],
    }

When running this code, the code editor renders with white, unhighlighted text, and I get this error in the terminal:
2024-06-11 05:45:36.228 ComponentRequestHandler: GET /workspace/dataplattformen/.venv/lib/python3.11/site-packages/code_editor/frontend/build/mode-sas.js read error
Traceback (most recent call last):
  File "/workspace/dataplattformen/.venv/lib/python3.11/site-packages/streamlit/web/server/component_request_handler.py", line 54, in get
    with open(abspath, "rb") as file:
         ^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/workspace/dataplattformen/.venv/lib/python3.11/site-packages/code_editor/frontend/build/mode-sas.js'

I noticed that sas is missing from the mode_list in __init__.py (line 102 in version 0.1.20). However, adding it did not resolve the issue.
Any ideas about what might be the cause? Thanks!

Hi @andersbenn
The end of the error you got:
FileNotFoundError: [Errno 2] No such file or directory: '/workspace/dataplattformen/.venv/lib/python3.11/site-packages/code_editor/frontend/build/mode-sas.js'

tells me that the mode file doesnt exist and indeed after checking, the only mode with those letters that I see is sass but there is no sas.
You can create one and add it either to your install or fork bouzidanas/ace-builds and add it to /src-noconflict. Make sure the file you add is called mode-sas.js and make sure to update ace-modules.d.ts and webpack-resolver.js


Issue with ImageNet Example

Hello,
I followed all the steps of the example including those in the issue, but my execution is getting stuck at a point. It is able to preprocess the training data but gets stuck in preprocessing the validation data. For the validation data, I downloaded the tar, untarred it, and repacked them into tars of around 250MB each, and then uploaded to S3. However, I am afraid I may have shuffled the order of images in this process.
Due to this or some other reason, the preprocessing step for validation data remains stuck indefinitely, however preprocessing of all the train data gets completed in 20 min. I am using 2 g2.8xlarge machines as slaves, which remain at high usage during the time the execution is stuck. Any tips on debugging or resolving this issue will be greatly appreciated. I have tried with both persist() and persist(StorageLevel.MEMORY_AND_DISK).
As shown below, it gets stuck after launching 26 tasks for the 25 tars of val data.
16/02/13 09:48:27 INFO scheduler.DAGScheduler: Submitting 26 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at mapPartitions at ScaleAndConvert.scala:31)
16/02/13 09:48:27 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 26 tasks
16/02/13 09:48:27 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, <IP_ADDRESS>, PROCESS_LOCAL, 2221 bytes)
16/02/13 09:48:27 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, <IP_ADDRESS>, PROCESS_LOCAL, 2228 bytes)
..
..
16/02/13 09:48:27 INFO scheduler.TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, <IP_ADDRESS>, PROCESS_LOCAL, 2228 bytes)
16/02/13 09:48:27 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on <IP_ADDRESS>:43854 (size: 1742.0 B, free: 27.4 GB)
16/02/13 09:48:27 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on <IP_ADDRESS>:38520 (size: 1742.0 B, free: 27.4 GB)
16/02/13 09:48:28 INFO storage.BlockManagerInfo: Added rdd_4_0 in memory on <IP_ADDRESS>:43854 (size: 24.0 B, free: 27.4 GB)
16/02/13 09:48:28 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1503 ms on <IP_ADDRESS> (1/26)

Also, on a related note I will like to point out 2-3 points that were not present in the Readme and which lead to errors on running the example.

Creation of a /tmp/spark-events/ directory is required else it leads to an error.
The readme states that the ~/.aws/credentials is required only on the master but not on the slaves. However, while running the example it is required on all the machines.
After changing the S3 bucket name, the Sparknet project needs to be rebuilt using 'sbt assembly'. This was not specified in the Readme. Further, the issue states that name of the validation folders is actually test, but in the version of the project present in the AMI, the names are actually val.txt and IL..._va

Hi Rahul, thanks for being so thorough! You're right about the omissions from the README. I'll add those and create an updated AMI.
Also, you're right that untarring and then retarring the validation files is a bit of a mess. I think we'll rewrite stuff to allow people to put ILSVRC2012_img_var.tar on S3 directly.
It's possible that the problem you're seeing is due to the way the validation files are tarred, but it's also possible that it's slow because of the size of the data. One way to check this is to replace the lines
var trainRDD = loader.apply(sc, "ILSVRC2012_img_train/", "train.txt")
val testRDD = loader.apply(sc, "ILSVRC2012_img_val/", "val.txt")

with
var trainRDD = loader.apply(sc, "ILSVRC2012_img_train/n01", "train.txt")
val testRDD = loader.apply(sc, "ILSVRC2012_img_val/validation.0", "val.txt")

or with
var trainRDD = loader.apply(sc, "ILSVRC2012_img_train/n015", "train.txt")
val testRDD = loader.apply(sc, "ILSVRC2012_img_val/validation.00", "val.txt")

When you use "ILSVRC2012_img_train/n01", that selects the training tar files whose filenames begin with n01 (I believe 127 out of 1000 tar files). If you use "ILSVRC2012_img_train/n015", it selects the training tar files whose filenames begin with n015, (13 out of 1000 tar files).
Similarly, "ILSVRC2012_img_val/validation.0" should select the validation tar files whose filenames begin with validation.0 (10 of them). And "ILSVRC2012_img_val/validation.00" should select the validation tar files whose filenames begin with validation.00 (only 1 of them).
This allows you to see if things work with much less data. If that works, then the problem has to do with the size of the data (and maybe using more nodes would help). If it fails, then the problem is somewhere else (possibly with the way the validation files are tarred).
Let me know how that goes!

Hello Robert,
Thank you for making the instructions easier.
About the problem I had initially put up, it was because I had created a tar of a directory. So in the ImageNetLoader code, if it is directory, it continues in a busy loop, this explains the high CPU usage on the nodes when the execution was stuck.
        while (entry != null) {           if (!entry.isDirectory) {             var offset = 0             var ret = 0             val content = new Array[Byte](entry.getSize().toInt)             while (ret >= 0 && offset != entry.getSize()) {               ret = tarStream.read(content, offset, content.length - offset)               if (ret >= 0) {                 offset += ret               }             }             // load the image data             val filename = Paths.get(entry.getName()).getFileName().toString             images += ((content, broadcastMap.value(filename)))             entry = tarStream.getNextTarEntry()           }         }
Thanks,
Rahul

However, after resolving that issue, I ran into multiple other issues. The initial AMI had the code at https://github.com/amplab/SparkNet/commit/0a6aaaa280b0d9f9118e15bebffe149b0dccb07f state and I also tried updating to this state https://github.com/amplab/SparkNet/commit/e5ee2c3caaa203ed614557ff5100f9d0fa5020ec  (which was latest till yesterday).
The train and val data pre processing steps went fine, the training also started but the reduce for testing got stuck. This happened multiple times, so I tried with both versions of the code. As you had suggested I limited train data to only those with prefix n01 (n015 prefix gives an error that n must be positive) and increased the number of g2.8xlarge slaves from2 to 5. Further I tried with java heap memory of 8GB and 32 GB. But in all these combinations, I was never able to complete the execution. Sometimes there were intermittent errors that data not serializable.
But the pertinent error was getting stuck in the reduce state of testing, and then one of the executors used to get lost even if the EC2 instance was in running state.
I am benchmarking the application to study its network properties, but unfortunately on multiple attempts, I was not able to pass beyond the first iteration of train, update and test.
I wanted to know whether I can use my already uploaded data to S3 using the new AMI and code or will it be better to reupload as per the new instruction steps.

Hi Rahul,
A couple questions/comments:

Have you tried the CifarApp? And if so, did that work? That one should be much easier to get running because you don't have to deal with S3. And if that doesn't work, it will be easier for us to track down the problem.
I'd suggest deleting all of the ImageNet data from S3 and uploading it again with the script SparkNet/scripts/put_imagenet_on_s3.py (this will take several hours, but at least it will ensure that we are dealing with the same directory structure and file names).
I'm not sure about the serialization error at the moment, but let me give it some thought.

Hello Robert,
Thank you for responding.
Thank you for your prompt response and patience.

Yes, CifarApp runs perfectly for me.
I will delete all the ImageNet data and start from scratch with new data and new AMI.
I am sorry I am giving  only vague details about the bug, but unfortunately, I lost my logs due to termination of the instances. So please ignore it for now, I will try as per Step 2, and report the status.
Thanks,
Rahul

OK, a small question - If we directly use the AMI do we still need to follow the steps in SETUP? I think I might have missed creating and attaching the instance volume to the slaves and so the nodes might be getting stuck when their memory gets full.
I was under the impression that those steps are not required, my bad. Do we also need to build caffe as per step 3?

Good point, we need to clarify this. If you're using the AMI, then you should only need to follow the instructions under "ImageNet" in the README. For now, I'd suggest using a fraction of the ImageNet dataset (e.g., 1% or 10%) so that you can ignore the part about mounting extra volumes. Once that's working, then we can use the full dataset (and that will probably require extra volumes or more memory).
You will probably also need to update the SparkNet repository on the AMI to the current master, and run "sbt assembly" again to recompile.
If you come across any error messages, please post them here!
We are working on simplifying the setup process, so hopefully that will be done soon and everything will get a lot easier.

Hello Robert,
I am using the new AMI now - ami-c0dd7db3.
After manually installing boto3 and pillow, I am trying to run the python script for uploading data but I am running into the following error. I tried when both the bucket was manually created and when it was not.
Command
python $SPARKNET_HOME/scripts/put_imagenet_on_s3.py sparknetdivideo \     --train_tar_file=~/data/ILSVRC2012_img_train.tar \     --val_tar_file=~/data/ILSVRC2012_img_val.tar \     --new_width=256 \     --new_height=256
`
Error
 Traceback (most recent call last):   File "/root/SparkNet//scripts/put_imagenet_on_s3.py", line 108, in <module>     upload_file("train.txt", train_label_file.read())   File "/root/SparkNet//scripts/put_imagenet_on_s3.py", line 39, in upload_file     s3.put_object(Bucket=args.s3_bucket, Key=targetname, Body=stream)   File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call     return self._make_api_call(operation_name, kwargs)   File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 396, in _make_api_call     operation_model, request_dict)   File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 111, in make_request     return self._send_request(request_dict, operation_model)   File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 140, in _send_request     success_response, exception):   File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 213, in _needs_retry     caught_exception=caught_exception)   File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 226, in emit     return self._emit(event_name, kwargs)   File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 209, in _emit     response = handler(**kwargs)   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 183, in __call__     if self._checker(attempts, response, caught_exception):   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 250, in __call__     caught_exception)   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 273, in _should_retry     return self._checker(attempt_number, response, caught_exception)   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 313, in __call__     caught_exception)   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 222, in __call__     return self._check_caught_exception(attempt_number, caught_exception)   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 355, in _check_caught_exception     raise caught_exception botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))

I've seen this error. I think it may have gone away after I did pip install aws --upgrade. Can you try that? Also perhaps try pip install boto3 --upgrade.

Ok, I will try that. But from the same machine,
aws s3 cp test s3://sparknetdivideo
is successful.

No help, same status.
root@ip-172-31-25-218:~/SparkNet# pip install boto3 --upgrade Requirement already up-to-date: boto3 in /usr/local/lib/python2.7/dist-packages Cleaning up...
root@ip-172-31-25-218:~/SparkNet# pip install aws --upgrade Requirement already up-to-date: aws in /usr/local/lib/python2.7/dist-packages Requirement already up-to-date: boto in /usr/local/lib/python2.7/dist-packages (from aws) Requirement already up-to-date: fabric>=1.6 in /usr/local/lib/python2.7/dist-packages (from aws) Requirement already up-to-date: prettytable>=0.7 in /usr/lib/python2.7/dist-packages (from aws) Cleaning up...
root@ip-172-31-25-218:~/SparkNet# python $SPARKNET_HOME/scripts/put_imagenet_on_s3.py sparknetdivideo1     --train_tar_file=~/data/ILSVRC2012_img_train.tar     --val_tar_file=~/data/ILSVRC2012_img_val.tar     --new_width=256     --new_height=256 starting to upload train.txt to bucket sparknetdivideo1 Traceback (most recent call last):   File "/root/SparkNet//scripts/put_imagenet_on_s3.py", line 108, in <module>     upload_file("train.txt", train_label_file.read())   File "/root/SparkNet//scripts/put_imagenet_on_s3.py", line 39, in upload_file     s3.put_object(Bucket=args.s3_bucket, Key=targetname, Body=stream)   File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call     return self._make_api_call(operation_name, kwargs)   File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 396, in _make_api_call     operation_model, request_dict)   File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 111, in make_request     return self._send_request(request_dict, operation_model)   File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 140, in _send_request     success_response, exception):   File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 213, in _needs_retry     caught_exception=caught_exception)   File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 226, in emit     return self._emit(event_name, kwargs)   File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 209, in _emit     response = handler(**kwargs)   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 183, in __call__     if self._checker(attempts, response, caught_exception):   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 250, in __call__     caught_exception)   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 273, in _should_retry     return self._checker(attempt_number, response, caught_exception)   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 313, in __call__     caught_exception)   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 222, in __call__     return self._check_caught_exception(attempt_number, caught_exception)   File "/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py", line 355, in _check_caught_exception     raise caught_exception botocore.vendored.requests.exceptions.ConnectionError: ('Connection aborted.', error(104, 'Connection reset by peer'))

If you feel this is my instance specific issue, please ignore it for now. I will try out a few things on my end.

Hmm.. I'll let you know if I think of something. I just experienced that problem yesterday and eventually got it to go away (but now I can't recreate it).
I thought it might be a problem with Anaconda, but it works for me with both Anaconda and the system's python.
Running pip show aws prints
Metadata-Version: 2.0
Name: boto3
Version: 1.2.3
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: UNKNOWN
License: Apache License 2.0
Location: /root/anaconda2/lib/python2.7/site-packages
Requires: jmespath, futures, botocore

Running pip show boto3 prints
Metadata-Version: 2.0
Name: boto3
Version: 1.2.3
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: UNKNOWN
License: Apache License 2.0
Location: /root/anaconda2/lib/python2.7/site-packages
Requires: jmespath, futures, botocore

I seem to have a old version of aws, but when upgraded, it shows that its up to date.
`root@ip-172-31-25-218:~/SparkNet/scripts# pip show aws
Name: aws
Version: 0.2.5
Location: /usr/local/lib/python2.7/dist-packages
Requires: boto, fabric, prettytable
root@ip-172-31-25-218:~/SparkNet/scripts# pip show boto3
Name: boto3
Version: 1.2.3
Location: /usr/local/lib/python2.7/dist-packages
Requires: botocore, jmespath, futures
root@ip-172-31-25-218:~/SparkNet/scripts# `
Secondly, I tried uploading files using boto3 to S3 from python console. It was successful in doing so. I think I am very near to solving my issue. Thanks for your help.

That's good to hear, keep us posted either way!

Ok, so instead of untarring the files in code, untarring it manually and passing the file handles i.e. f in
f = open("test.txt)
as Stream in s3.put_object seemed to solve the issue atleast for train and test.txt. Now the script has is running, so I guess it should run to completion.

Cool. I'm interested in knowing if it runs to completion.
This'll be helpful if more people run into this error, in which case we'll need to robustify the script against this kind of thing.

Ok, so I had left it running overnight. The script finished uploading the last val tar at around 3.00 AM PST. Today, I will go ahead and run the actual training program.

Awesome, let me know how that goes!

Cluster creation command
SparkNet/ec2/spark-ec2 --key-pair=newtrial --identity-file=newtrial.pem --region=eu-west-1 --zone=eu-west-1c --instance-type=g2.8xlarge --ami=ami-c0dd7db3 -s 5 --copy-aws-credentials --spark-version 1.5.0 --no-ganglia --user-data SparkNet/ec2/cloud-config.txt launch sparknet
Job Submit Command
/root/spark/bin/spark-submit --class apps.ImageNetApp $SPARKNET_HOME/target/scala-2.10/sparknet-assembly-0.1-SNAPSHOT.jar 5 sparknetdivideo > log.txt 2> log.txt
Error
16/02/16 20:05:26 WARN scheduler.TaskSetManager: Lost task 3.0 in stage 9.0 (TID 64, <IP_ADDRESS>): java.lang.IllegalArgumentException: n must be positive         at java.util.Random.nextInt(Random.java:300)         at scala.util.Random.nextInt(Random.scala:65)         at libs.MinibatchSampler.<init>(MinibatchSampler.scala:18)         at apps.ImageNetApp$$anonfun$main$3.apply(ImageNetApp.scala:181)         at apps.ImageNetApp$$anonfun$main$3.apply(ImageNetApp.scala:157)         at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:99)         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)         at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)         at org.apache.spark.scheduler.Task.run(Task.scala:88)         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)

I tried adding two debug statements and divided the syncInterval by 2 before passing it to the MinibatchSampler and now getting another error.
log("Minbatch sampler len", len)  log("Minbatch sampler syncinterval", syncInterval)  val minibatchSampler = new MinibatchSampler(trainMinibatchIt, len, syncInterval/2)
 Exception in thread "main" org.apache.spark.SparkException: Task not serializable         at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)         at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)         at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)         at org.apache.spark.SparkContext.clean(SparkContext.scala:2021)         at org.apache.spark.rdd.RDD$$anonfun$zipPartitions$1.apply(RDD.scala:849)         at org.apache.spark.rdd.RDD$$anonfun$zipPartitions$1.apply(RDD.scala:849)         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)         at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)         at org.apache.spark.rdd.RDD.zipPartitions(RDD.scala:848)         at org.apache.spark.rdd.RDD$$anonfun$zipPartitions$2.apply(RDD.scala:855)         at org.apache.spark.rdd.RDD$$anonfun$zipPartitions$2.apply(RDD.scala:855)         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)         at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)         at org.apache.spark.rdd.RDD.zipPartitions(RDD.scala:854)         at apps.ImageNetApp$.main(ImageNetApp.scala:156)         at apps.ImageNetApp.main(ImageNetApp.scala)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)         at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)         at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.io.NotSerializableException: java.io.PrintWriter Serialization stack:         - object not serializable (class: java.io.PrintWriter, value: java.io.PrintWriter@1171b886)         - field (class: apps.ImageNetApp$$anonfun$main$3, name: trainingLog$1, type: class java.io.PrintWriter)         - object (class apps.ImageNetApp$$anonfun$main$3, <function2>)         at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)         at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)   at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)         at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)         at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:84)         at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)         ... 26 more

Ah, I believe the key is in the line
java.lang.IllegalArgumentException: n must be positive at java.util.Random.nextInt(Random.java:300)

This is happening within line 181 in ImageNetApp.scala, reproduced here:
val minibatchSampler = new MinibatchSampler(trainMinibatchIt, len, syncInterval)

We are trying to sample syncInterval = 50 minibatches from a partition which has len minibatches. In this case, because you are probably using 1% of the ImageNet data, len is probably smaller than 50, so we end up calling Random.nextInt with a negative number. So this should be fixable either by making syncInterval smaller or by using more data. You can use more data by setting
    var trainRDD = loader.apply(sc, "ILSVRC2012_img_train/train.000", "train.txt")

instead of
    var trainRDD = loader.apply(sc, "ILSVRC2012_img_train/train.0000", "train.txt")

As an aside, a bunch of this code is going to change. In particular, MinibatchSampler will go away.

The function log can't be put in a map call (it writes to a file that lives on the Spark master). You can use a regular print statement, and the output will appear on the worker logs at http://<IP_ADDRESS>:8080 (IP address taken from the screenshot you posted).

Oh sorry for the second issue I just wanted to give as much info as possible.
About the first  issue about n, I was playing with this all day on Sunday. It was just that I did not see that the version in Git itself uses a subset of data. Since I had not made that change in my local code, I was assuming it used all the code, and so I thought the error might be elsewhere.

The place where we pull a subset of ImageNet is the line
var trainRDD = loader.apply(sc, "ILSVRC2012_img_train/train.0000", "train.txt")

Basically, the script scripts/put_imagenet_on_s3.py creates 1000 tar files on S3 named
ILSVRC2012_img_train/train.00000 through ILSVRC2012_img_train/train.00999. The function loader.apply(sc, "XXX", "train.txt") pulls all of the files with the prefix XXX. This makes it easy to use 1% or 10% of the data.

Yes yes,  I was aware of that. I missed looking at it when I got the error that n must be positive.
Ok for now the training is working successfully.  For training with the entire data set, do you feel that adding a volume to the slaves is necessary? If yes, does it have to be converted into ext4 before mounting as per the instructions on Amazon's documentation.
For reference
0.002: loading train data 6.823: loading test data 7.456: processing train data 22.104: numTrainMinibatches = 500 22.104: processing test data 27.208: numTestMinibatches = 200 27.208: computing mean image 29.202: coalescing 29.321: trainPartitionSizes = Array(100, 100, 100, 100, 100) 29.362: testPartitionSizes = Array(40, 40, 40, 40, 40) 57.879, i = 0: broadcasting weights 58.576, i = 0: setting weights on workers 62.156, i = 0: testing 70.286, i = 0: 0.08% accuracy 70.286, i = 0: training 176.556, i = 0: collecting weights 192.708, i = 1: broadcasting weights 193.422, i = 1: setting weights on workers 196.24, i = 1: training 303.133, i = 1: collecting weights 318.766, i = 2: broadcasting weights 319.4, i = 2: setting weights on workers 322.408, i = 2: training 429.262, i = 2: collecting weights 445.211, i = 3: broadcasting weights 445.834, i = 3: setting weights on workers 448.604, i = 3: training

I'd try it with 10% of the data and make sure that works fine.
I wouldn't bother adding volumes because Spark can be extremely slow when reading RDDs from disk. The best solution in that case might be to leave the JPEGs in the RDD (that is, don't decompress them), and then decompress them as needed for training. That might enable everything to fit in memory.

Hello Robert,
Yes I tried to do with the whole data after increasing the allocated memory to 50G.
val conf = new SparkConf()                                                .setAppName("ImageNet")                                          .set("spark.driver.maxResultSize", "30G")                        .set("spark.driver.maxResultSize", "50G")                        .set("spark.task.maxFailures", "1")                                   val sc = new SparkContext(conf)
But it has got stuck in this phase

I will try out your suggested option of keeping the JPEGs in compressed format, and let you know what happens.
Thank you for your patience and support. I apologize if I have asked too many silly questions.

Ok so finally it has crashed. So leaving in the compressed format is the way to go I guess.
16/02/16 21:26:48 ERROR client.TransportResponseHandler: Still have 1 requests outstanding when connection from /<IP_ADDRESS>:47135 is closed 16/02/16 21:26:48 ERROR shuffle.RetryingBlockFetcher: Failed to fetch block taskresult_2066, and will not retry (3 retries) java.io.IOException: Connection from /<IP_ADDRESS>:47135 closed         at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:104)         at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:91)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)         at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)         at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)         at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)         at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:739)         at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:659)         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)         at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)         at java.lang.Thread.run(Thread.java:745) 16/02/16 21:26:48 ERROR scheduler.TaskResultGetter: Exception while getting task result java.io.IOException: Connection from /<IP_ADDRESS>:47135 closed         at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:104)         at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:91)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)         at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)         at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)

Hi Rahul, we just made a bunch of new changes to the code base, hopefully simplifying a number of things (see #72). There are a couple of consequences for ImageNetApp in the meantime.

The code is simpler, and most of the clunky classes are gone (e.g., MiniBatchSampler).
Training will be slow (we had to disable GPUs due to a bug in javacpp javacpp-presets/issues#147), but we plan on fixing this soon. This will make it difficult to train ImageNet to high accuracies (though it should still run).

Once I change the ImageNetApp to have the DataFrames store compressed images, training the full ImageNet dataset should work (right now it should work fine with 10% of the training data and all of the validation data).

Hello Robert,
Thank you for the update.
I have subscribed to all the updates of this Github project, and also want to ask some architectural/systems related questions about the proposed solution in #71.
Thanks,
Rahul

Great! Let's discuss that in #71.

Closing because the issues raised here are being discussed in separate issues. Feel free to reopen if something is unaddressed.


check IsNotExist error when validating mountpoint

This fixes #9952

v1.0 triage needed: Marked the associated issue #9952 as v1.0-candidate

@k8s-bot ok to test


Warn when a package is possibly skipped

(with an option to make this a hard error)
We currently require that the root crate has dependencies on all other local crates so that we can find them in the lockfile. This is simple, but has the problem of silently skipping crates/packages that are not dependencies of the main crate. We should at least warn when this happens, but I'm not sure how to do that correctly.

Since the Cargo.lock stores the package name, we could simple compare it to the list of local packages we found. Not hard to do at all (except if I'm missing something)!
(there could of course be duplicate packages, we should warn in that case too)

I did miss something: The Cargo.lock may not exist -.-
The Cargo.toml, however, should.

This is done


Add support for YOLO to output bounding boxes

Implemented option for users to choose between two output formats: byte conversion of the image with drawn bounding boxes or a list of bounding boxes for further processing.
Also, based on this: https://onnxruntime.ai/docs/tutorials/mobile/pose-detection.html, if we didn't modify yolo_detection(), how can it can return two values (rawImage and boxOutput)? It looks like the used yolo_detection() from yolo_e2e.py is ahead from what I used, any ideas?
val rawOutput = (output?.get(0)?.value) as ByteArray
val boxOutput = (output?.get(1)?.value) as Array<FloatArray> // Index out of bounds

Also, based on this: https://onnxruntime.ai/docs/tutorials/mobile/pose-detection.html, if we didn't modify yolo_detection(), how can it can return two values (rawImage and boxOutput)? It looks like the used yolo_detection() from yolo_e2e.py is ahead from what I used, any ideas?

I'm not quite sure what you're asking here and whether it's about the pose model or the object detection model. Each has different output. And can you clarify what 'rawImage' equates to?

/azp run onnxruntime-extensions.CI,licence/cla

Also, based on this: https://onnxruntime.ai/docs/tutorials/mobile/pose-detection.html, if we didn't modify yolo_detection(), how can it can return two values (rawImage and boxOutput)? It looks like the used yolo_detection() from yolo_e2e.py is ahead from what I used, any ideas?

I'm not quite sure what you're asking here and whether it's about the pose model or the object detection model. Each has different output. And can you clarify what 'rawImage' equates to?

Sorry, it's for object detection model, and the rawImage I am referring is the detected image with drawn bounding boxes.

You can add an Identity step at the end to produce an additional model output from the ScaleBoundingBoxes step.
e.g. after the last post-processing step, add an Identity step like below
        # Encode to jpg/png
        ConvertBGRToImage(image_format=output_format),

        # also return bounding boxes
        (Identity(name="bounding_boxes"),
         [utils.IoMapEntry("ScaleBoundingBoxes", producer_idx=0, consumer_idx=0)]),


fix: missing images from chart dependencies

Signed-off-by: Harry<EMAIL_ADDRESS>
#1973

No need to manually do this. You can check out codes at L101-L112

As we discussed, no changes need to be made.


Use the milli's heed dependency

Pull Request
What does this PR do?
Fixes #2210

This PR (https://github.com/meilisearch/milli/pull/464) is to be merged before approving this.
PR checklist
Please check if your PR fulfills the following requirements:

[x] Does this PR fix an existing issue?
[x] Have you read the contributing guidelines?
[x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!

Resolved merge conflicts in PR

hey @psvnlsaikumar ! Thanks so much for contributing! Could you squash all your commits into one please?

@psvnlsaikumar, I need this merged ASAP, so what I am gonna do is merge this branch into another branch, make my fixes there, and merge it in main afterward. This way your work will be present in main, while I get my merge on time :)
Thanks again for your contribution :)

Sure @MarinPostma . That works. I was going to get the work done tonight. But if you are going to take care of it then sure. FYI, the version of milli is to updated once we have the newer version or milli with the recent changes( export of heed) available.

@psvnlsaikumar  yes, that should be done this afternoon