Connect more than one camera

I connected the PS move and the Eye and they worked great, but i need to connect more than 1 camera, i used LibUSB to identify the PS Eye, i use unity 5.3 and windows 8.1 64bit

PSMoveAPI (the library psmove-unity5 uses under the hood) technically supports multiple cameras see psmove_tracker_new_with_camera_and_settings, however psmove-unity5 doesn't currently attempt to open multiple cameras, see PSMoveManager.cs. However, that wouldn't be hard to change that part. Basically we could just add a camera count parameter to the psmove manager and attempt to open multiple trackers if the count is greater than 1. That said, once you have two trackers active you need to decide how to reconcile the positions you get back from each tracker (average them? Compute an intersection from each camera frustum?). Also the DK2 coregistration tool would need to be modified to support co-registration for each camera as well (right now it assumes only one camera). Unfortunately I won't have time to make this addition until after March 20th as I'm currently swamped with prep for the Game Developers Conference.
Also @rovarma I recall you're doing work with multiple cameras in psmoveapi. Is there anything else I'm going to run into when adding support for this.

Not from the psmoveapi side; it's indeed as simple as opening the cameras with psmove_tracker_new_with_camera_and_settings.

thanks guys i made 2 cameras work at the same time but still i need to work on averaging the position of the PS move between the trackers.
I did it by adding a new function from the DLL in the PSMoveAPI.cs script :
[DllImport("psmoveapi_tracker.dll")]
public static extern IntPtr psmove_tracker_new_with_camera(int camera);
I called this function in the PSMoveManager script instead of psmove_tracker_new_settings(ref settings) in the function WorkerContextSetupTracking.
I turned the PSMoveTracker value in the WorkerContext Class in the PSMoveManager script into an array of IntPtr to allow more than 1 camera to be used.
You pass the index of the camera you need to be working to psmove_tracker_new_with_camera.
And don't forget to close the cameras in the function WorkerContextTeardownTracking.

@hazem17 Cool! Glad to hear exposing that API worked.
It also occurs to me that the co-registration transform (says where the PS3EYE camera is relative to the DK2 headset) that's computed by the visual coregistration tool is only valid for the first camera. The co-registration tool uses psmove_tracker_new_with_settings when setting up the tracker, which defaults to camera 0. If you want to get a valid tracking position for the second camera, a camera index needs to be passed into the co-registration tool. The psmove_fusion API also needs to be updated to support more than one transform see psmove_fusion.cpp.

@HipsterSloth I have a question, what is the functionality of PSMoveFusion?, cause i couldn't find any documentation for it.

The functions for PSMoveFusion are documented here: https://github.com/HipsterSloth/psmoveapi/blob/psmove_unity5/include/psmove_fusion.h
The functions there are used to put the raw ps move location into the space of another tracking camera. In particular, psmove_fusion_get_transformed_location, is used to transform the raw psmove position in the space of the ps3eye into the space of the DK2 tracking camera. The transform that does this is computed by the co-registration tool. psmove_fusion_new loads the coregistration transform saved out by the coregistration tool.

@HipsterSloth thanks again, can i get the current image of the eye for debug purpose?

If you run the ps3eye_sdl.exe in the x86_64 folder you can see what the camera sees. But that will only work when you don't have your unity application running with tracking active (since only one application can connect to the camera at the same time). If you want to display the camera image in Unity i'd need to expose the functionality. (Psmoveapi has a "psmove_tracker_get_image" function that returns an rgb frame of the current image)

@HipsterSloth If you will have time, are you going to add multiple cameras support to your plugin?

Yes it is my plan to. I had originally intended to do with the replacement for psmoveapi I was working on with cboulay. We've been making a windows service/unix daemon that manages the psmoves and tracking cameras in one nice unified manner with a config tool front end (similar to the visual coregistration tool checked in now). We wanted to add new functionality to this service like multi-camera support and tracking prediction. However, it's taking us a while to get the service to the point where is will be usable by psmove-unity5 (probably at least another month). Since there have been a few people that have asked me about this now, I think I'll make the multi-camera stuff work using the existing psmoveapi for the time being since that's a lot less work (then port it over to the new service).
@hazem17 How is the multi-camera averaging work going on your end? Did you still need help getting the co-registration tool to work with multiple cameras? I was going to try and make the multiple camera setup work this week. If there is code you would like integrated into psmoveapi/psmove-unity5 I can setup a branches for you to check this in to.

Ok, great. I'll wait. First I need to buy second camera :) Do you think it is possible to achieve 360 tracking with two cams or should it be like 3-4 cams?

The ps3eye has an field of view of 75 degrees and a range of about 6-7 feet (as compared to the 100 deg FOV, 15ft range for the Oculus Sensor, see https://imgur.com/a/0YcNA) so two ps3eye camera's might be ok for 360 degree tracking. But having not tried this configuration myself I couldn't say. At more than two cameras I would start worrying about saturating the USB bus's ability to transfer the feeds reliably at 60fps.
@rovarma In your experiments with multiple ps3eye cameras are two ps3eye's sufficient for 360 degree tracking? How you tried more than two cameras?

I finally started work on this tonight. My approach is going to be similar to what the VrTracker project is doing and use OpenCV's triangulatePoints method to find the psmove position using the extrinsic camera parameters (i.e. the PS3EYE transforms relative to the Oculus DK2 tracker) generated from the coregistration tool and the 2d tracking position from each tracker. This approach will initially be implemented using two cameras (I just got my dual camera rig setup at home yesterday). If this works then I'll try extended this to work with more than two. Probably something along the lines of "every frame pick the two cameras where the psmove has the largest tracking blobs and then use triangulate on those". I'm currently working on making the coregistration tool support multiple cameras. Once that's done I'll update the fusion API to support computing a position using an array of trackers. Then I can update psmove-unity5 to use this new API. Hopefully I can get a version of this working by this weekend. I'll keep this thread posted with updates.

Here is what i made so far in the multiple camera : https://www.mediafire.com/?094qpxd2wgpaaz0
I know it's not much, but i hope it helps.

@Czeus @hazem17 I just got the multi-camera triangulation working in the visual coregistration tool:
https://www.youtube.com/watch?v=HT5moG_qRpM
There are a few changes to the psmove_fusion api that expose this functionality. After I finish testing the psmoveapi changes, I'll make new builds of that and then integrate this into psmove-unity5. Fortunately that part should be pretty straightforward. I'll let you guys know once that is checked in. This work will go into the new multicam branch for you guys to test.

Your video is private, I cant open it. Im looking forward to your build ;)

Whoops! I just set it to public now.

Great work, is there a limit for the cameras to use?, also when will the unity package be available?

@hazem17 Right now there is a hard coded limit of 3 cameras. I have personally only tested with two cameras so far. Theoretically there is no reason that number couldn't be larger. However I suspect that you can probably only have at most 2-3 cameras per usb bus before the bus will become saturated. How many cameras were you thinking of using? I ask because I want to test on my machine a reasonable maximum is. I'm hoping to get the first version of this integrated with the Unity package by this weekend. I'll make a post here when it's ready along with setup instructions.

We have 5 Cameras here. :smile:

I has thinking about using 8 cameras at the same time.

@MHDante @hazem17 I'm going to test on my PC this weekend with 5 PS3Eye cameras (that's all I can get ahold of at the moment). That said if you guys are planning on using that many cameras you'll almost certainly need one or two usb extension cards:
http://www.amazon.com/gp/product/B00FPIMJEW
The other major issue is that the coregistration tool that is used to figure out the ps3eye camera location relative to the DK2 tracker is ultimately limited to the DK2 tracking volume range. So trying to make a PS3EYE tracking volume any larger than that currently won't work. It should be possible to do the same thing with the Oculus CV1 which has a larger tracking volume, but psmoveapi isn't compiled against the 1.3 SDK yet. I assume you guys were both using the DK2?

I don't want to use Oculus, i just want to track the PS moves in the largest area possible, can it be achieved?

@MHDante @hazem17 @Czeus
Ok guys, I have all this worked checked into the multicam branch and it's ready for you guys to test. I was able to successfully test this with 5 ps3eye camera connected at the same time BUT there are some important caveats that make this kind of an unfriendly experience for more than two cameras at once. (To be honest, I was quite shocked this would work at all with more than two cameras at all).
Caveat 0: When dealing with more than two PS3EYE cameras it is very easy to saturate the USB bus they are connected to.  The ps3eye_sdl tool now takes a camera index. Use that first to make sure you can get video feeds from all cameras at once.
Caveat 1: Right not the blink calibration is done one camera at a time, back to back. When you have just one or two cameras, the calibration wait isn't too bad. Once you get to 5 or more cameras it gets kind of obnoxiously long. It's also trickier to try and keep a psmove in view of many cameras, especially when some of them are behind you. This could be improved, but it would take some refactoring of the blink calibration code, which will take a while.
Caveat 2: The visual coregistration process is a lot more painful with more than two cameras. For the same reason as the blink calibration. It's difficult to keep the DK2 and the psmove in view of all of the cameras AND move both around smoothly. Technically you don't have to keep them view the whole time, but I've found you don't get a camera transform if you have gaps in your tracking. Really this process needs to refactored to use OpenCV's cv2.solvePnP method that correlates known points on the ground to tracker location on the screen. The VrTracker library uses this approach. This also will take a while to implement
Caveat 3: Triangulation works between a single pair of cameras. When there are more than two cameras I pick the first two cameras that can see the controller (see here). While this is the simplest solution this can lead to jerky results when we hop between pairs of camera if the camera poses computed during coregistration aren't aligned accurately (which they aren't right now due to the method being used).
So, please go ahead and give this a shot. I'm going to write some more detail setup instructions and add those to the wiki tomorrow (it's currently 3AM now and I've given this day all I can give).

@hazem17 Currently the visual co registration tool relies on a DK2 to correlate points in DK2 tracking volume to ps3eye tracked positions and allow us to figure out where the PS3EYE cameras are relative to each other. It is presently not possible to make multicamera tracking work without a DK2 attached for the visual coregistration tool. After the coregistration tool saves out the PS3EYE camera poses you technically don't need the DK2 attached anymore. I want to replace this requirement with a different calibration method that doesn't require the DK2 (see caveat 2 from my previous post). But that will take me at least another week to implement.
@MHDante @hazem17 @Czeus
Also if you guys want to add functionality to my fork of psmoveapi that's used in psmove_unity5 feel free. You'll notice that there is a new third_party/clone_psmoveapi.bat script. That will pull down everything you need to build the psmoveapi dependencies yourself. So let's say you wanted to change the implementation of one of the psmove_fusion methods, you would do the following:

Go to third_party folder
Run clone_psmoveapi.bat
Go to psmoveapi folder
Run InitialSetupX64.bat (or InitialSetupWin32.bat if building 32-bit dependencies)
Run GenerateProjectFilesX64.bat
Go to build folder
Open PSMOVEAPI.sln in MSVC 2013
Make your psmoveapi code changes
Select the Release|x64 configuration
Build solution
Return back to the root project directory
Go to Assets\Plugins\x86_64\scripts
Run BindPSMoveAPIBuild.bat

Select the third_party\psmoveapi folder
Points localally created SetPSMoveBuildVars.bat to that folder

Run CopyPSMoveAPIFiles.bat
psmove-unity5 now has your updated version of psmoveapi. Now you can test your changes in unity.

@hazem17 I have an idea that will address both your wish to not have to use a DK2 and should improve the the poor camera pose estimation problem I mentioned with caveat 2.
We need a way to determine what each PS3EYE camera pose (position & orientation) is relative to some user defined origin. We could do define a tracking origin using a calibration pattern with a known dimension and then set the psmove controller at each position of the calibration target and record the position. A calibration target would look something like this (though probably with only 4 or 5 points):

So to calibrate your tracking space you would do the following:

Print out the calibration pattern (making sure not to resize it)
Set the calibration pattern down on the floor (or table) in view of all of the tracking cameras
Run the modified visual coregistration tool
Set the psmove controller on maker position 1
Wait for the tool to record the psmove controller position
Repeat for the next 4 or 5 calibration points
[optional] If you have a DK2 connected, set it in the middle of the calibration pattern
[optional] Wait for the tool to record your DK2 position
Close the calibration tool and launch unity

Since this process gives me the 2d pixel position of the controller at each marker location and I already know the 3d position of each marker location (since I get to define that with the marker pattern) I can then use OpenCV's cv2.solvePnP to compute the camera pose. The camera pose I get form this method should be a lot more reliable than the old method.
My questions to you:

Are you planning on placing all of your PS3EYE cameras such that they can look down on a flat surface (floor or table)?
Will it be the possible for all PS3EYE cameras to see the origin pattern on the floor or a table? (this wouldn't be possible if you have a huge room or were trying to make this work across multiple rooms)?

This plan sounds like it shouldn't be too hard since I already have most of the pieces I need in the existing visual coregistration tool. I was planning on trying to implement it tomorrow night. I'll post an update tomorrow depending on how far I get.

I want to set the cameras so they can fully cover a room, some of the cameras won't look at the same target, is it possible for the calibration to do that.

@hazem17 I could make that work if I added the ability to specify an offset for the calibration target. You could then turn one set of cameras, calibrate against the first target (with no offset), then turn on the next set of cameras and off the first set of cameras, and calibrate against the target with an offset you specify. You would have to carefully measure the x and y offsets with a tape measure. This assumes that the targets are on the same surface (floor or table). I can add a vertical offset too if that's not the case, but that might be harder to measure.

@HipsterSloth I tried the multicam build at our PCs here. It seems that our PCs only have 2 root Hubs.:( I'll submit a request for an expansion card in next year's budget.

@HipsterSloth can i choose which camera calculates the PS move current position from unity, i was thinking that i can calculate which camera has the biggest radius of the PS move and is currently tracking it and put the offset manually for each camera.
For example if i placed three cameras beside each other and made the middle one calculates the PS move normally, the right one adds offset to the x axis and the left one subtracts offset from x axis, so if the PS move passes in front of specific camera this one becomes the main tracking camera and adds the offset to the PS move.
I know it will be very hard coded but i think it will fit my target.

@hazem17 You should be able to chose which position to use if you switch PSMoveWorker.ControllerUpdatePositions() from using PSMoveAPI.psmove_fusion_get_multicam_tracking_space_location() back over to using PSMoveAPI.psmove_fusion_get_tracker_pov_location() per tracker, and then use the z-value (since it will be proportional to the radius in that case.
As for the script errors, it sounds like you need to download visual studio community 2013 (http://go.microsoft.com/fwlink/?LinkId=517284) and cmake (https://cmake.org/files/v3.5/cmake-3.5.2-win32-x86.msi) if you want to build psmove api using those build scripts.
Finally I have made some progress on the new multi camera calibration method that doesn't need a DK2 tracker (calibration pattern instead):
https://youtu.be/33cWRaCC9hU
I'm still fixing a few bugs with it but I'm hoping to have a build of that uploaded by this Wednesday. I'll post here when it's available.

While I was able to fix some bugs with how the camera intrinsic matrix was being computed (encodes camera horizonal and vertical FOV) I still haven't been able to get the new multi-camera calibration tool completely working. After pounding my head against it for a few days I was able to get the correct camera orientation from cv::solvePnP, but I still can't get the camera position right. I went ahead and checked in the changes I have so far to psmoveapi, including the new calibration tool source, but I haven't bothered to make a new build of psmove-unity5 yet since the tool doesn't fully work yet. In case any of you are curious you can check out the source for it here:
https://github.com/HipsterSloth/psmoveapi/blob/psmove_unity5/src/utils/visual_setup/visual_tracker_setup.cpp
I'll be traveling for the next 4 days. When I get back next week hopefully some fresh eyes will reveal what is going on with the camera position calculation.

Just came upon this discussion here and I have to say your work is amazing! I'll look into the code a bit and hopefully can contribute. Just wondering, do you think it would ever be possible for this to work with a large amount of cameras, such as 128 as used in the Zero Latency system?

I just got the multicam setup tool working and the last of the fixes checked into my fork of psmoveapi. This includes getting it working with the Oculus 1.3 SDK (though you don't have to use the oculus if you don't want to, it's an optional step in the setup). I need to update the test unity project to use the latest OVR plugin for unity. I'll get to that this weekend along with writing up some documentation and a video about how to use the new tool.

@fanfreeman Thanks! I do what I can (along with the help of many others). Any help is always appreciated. In the current setup of this library, the most cameras you can realistically connect to a PC is 5. In order to go past this limit you would need many machines, each running their own psmoveapi server that would network their tracking info to the client doing the rendering. I am separately working on a psmove service that networks tracking data to a client, but the networking in that case is really just an interprocess communication mechanism. While it might be theoretically possible to extend this service into something that would work on a larger scale, it would be a lot of work. My goal is to get the first version of psmoveservice working and integrated with SteamVR. After that if someone wanted to extend the service to a multi server tracking system, they would be more than welcome to, but I that will be a few months out.

@MHDante @hazem17 @Czeus
I just updated the multicam branch of psmove-unity5 with the new visual_tracker_setup.exe camera setup tool and upgraded psmoveapi to use the Oculus1.3 SDK. I updated the project to Unity 5.3.4.p1 so that I could support the Oculus 1.3 SDK. This also requires installing the new Unity OVRPlugin from Oculus. I didn't bother building the 32-bit version of psmoveapi in this branch. Let me know if you are testing against these.
In order to use the new visual_tracker_setup.exe tool you'll need to print out the calibration mat:

Download the Calibration Mat PDF for the best print quality.
The setup steps are then as follows:

Place the calibration mat on a flat surface in view of all of the PS3Eye cameras
This will define the origin of your PS3EYE tracking volume space
Launch the magnetometer_calibration.exe calibration tool
On the second step where you set the controller down on a flat surface, place the controller in the center of the calibration map making sure the red +x axis points along the "start" button on the psmove controller and the blue +z axis points away from the front of the controller
[optional] If you plan on using psmove-unity5 with the oculus HMD make sure it's connected and turned on now.
Launch the visual_tracker_setup.exe tool
Follow the on screen calibration instructions
In particular, when it asks you to place the psmove on the calibration points, stand the controller vertically on the corner of the labeled point on the calibration mat.
[optional] If you connected an OculusHMD the last step will ask you to place the HMD on the calibration mat with the front edge aligned with the black square brackets. This is so that calibration tool knows where the PS3EYE origin is in the space of the Oculus tracking camera. The PS3EYE camera transforms will be saved out relative to the Oculus tracking camera.
The tool will save the camera transforms in C:\Users\<username>\AppData\Roaming\.psmoveapi\camera_pose_libusb_b<usb_bus>_p<usb_port path>.csv, where <usb_bus> and <usb_port_path> uniquely identify the camera based on the USB port you have plugged in
IMPORTANT: If you change the port a camera is plugged into you need to run the visual_tracker_setup.exe calibration tool again since the cameras are identified by usb port (PS3EYE cameras don't have a unique identifier).

You can still run the old visual_coregister_dk2 tool if you want, but that tool requires you to have a DK2 attached, where the new visual_tracker_setup.exe does not.
Let me now if you guys have any questions or run into any issues.

Any chance to see a working OSX branch for 10.11.4 ?
Anybody could compile the XCode .bundle to work with Unity ?
I could compile the library and examples are working, I tried compile with -XCode parameter to obtain the xcode project, but after fixing some compile error, I have link error complaining _SetCameraExposure is not X86_64, any clue ?
I already tried other PSMOVE_UNITY5 OSX brenches, but no luck, Unity doesn't find the library.
I also tried to copy my .a .dylib and .dll compiled fresh from latest PS_MOVE_API release, but no luck.
Did anyone have any luck with Unity on the Mac ?

Hi im running the @hazem17  build and works fine with multiple cameras. I will try the step by step calibration by @Hipersloth to test the official branch.


Small change required to successfully use XTL with nvcc CUDA compiler

Hi,
I was hoping that since XTL works with any C++14 compiler, the compilation of host only CUDA 10 code should work fine with XTL. Normally, I would not attempt this and instead find a replacement that is out-of-the-box compatible with CUDA. However, I need to write some CUDA as part of a large codebase which uses xtensor, openmc. It does seem that xtensor isn't far off from being able to compile host code with nvcc, I only obtain the following errors when compiling the most basic of xtensor programs (the first one on its readthedocs):
gavin@gpad:~/scratch$ nvcc xtensor_example.cu
/usr/local/include/xtl/xsequence.hpp:136:87: error: expected primary-expression before ‘unsigned’
         struct sequence_forwarder_impl<R, A, void_t<decltype(std::declval<R>().resize(std::size_t()))>>
                                                                                ^~~~~~~~
/usr/local/include/xtl/xvariant_impl.hpp: In function ‘constexpr decltype(auto) mpark::visit(Visitor&&, Vs&& ...)’:
/usr/local/include/xtl/xvariant_impl.hpp:2676:96: error: parameter packs not expanded with ‘...’:
     return (detail::all({!vs.valueless_by_exception()...})
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                     ^
/usr/local/include/xtl/xvariant_impl.hpp:2676:96: note:         ‘vs’
/usr/local/include/xtensor/xstrided_view_base.hpp: In instantiation of ‘void xt::detail::strided_view_args<adj_strides_policy>::fill_args(const S&, ST&&, std::size_t, xt::layout_type, const V&) [with S = xt::svector<long unsigned int, 4, std::allocator<long unsigned int>, true>; ST = const xt::svector<long int, 4, std::allocator<long int>, true>&; V = std::vector<mpark::variant<long int, xt::xrange_adaptor<xt::placeholders::xtuph, long int, long int>, xt::xrange_adaptor<long int, xt::placeholders::xtuph, long int>, xt::xrange_adaptor<long int, long int, xt::placeholders::xtuph>, xt::xrange_adaptor<long int, xt::placeholders::xtuph, xt::placeholders::xtuph>, xt::xrange_adaptor<xt::placeholders::xtuph, long int, xt::placeholders::xtuph>, xt::xrange_adaptor<xt::placeholders::xtuph, xt::placeholders::xtuph, long int>, xt::xrange_adaptor<long int, long int, long int>, xt::xrange_adaptor<xt::placeholders::xtuph, xt::placeholders::xtuph, xt::placeholders::xtuph>, xt::xrange<long int>, xt::xstepped_range<long int>, xt::xall_tag, xt::xellipsis_tag, xt::xnewaxis_tag> >; adj_strides_policy = xt::detail::no_adj_strides_policy; std::size_t = long unsigned int]’:
/usr/local/include/xtensor/xstrided_view.hpp:593:1:   required from ‘auto xt::strided_view(E&&, const xstrided_slice_vector&) [with E = const xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::svector<long unsigned int, 4, std::allocator<long unsigned int>, true>, xt::xtensor_expression_tag>&; xt::xstrided_slice_vector = std::vector<mpark::variant<long int, xt::xrange_adaptor<xt::placeholders::xtuph, long int, long int>, xt::xrange_adaptor<long int, xt::placeholders::xtuph, long int>, xt::xrange_adaptor<long int, long int, xt::placeholders::xtuph>, xt::xrange_adaptor<long int, xt::placeholders::xtuph, xt::placeholders::xtuph>, xt::xrange_adaptor<xt::placeholders::xtuph, long int, xt::placeholders::xtuph>, xt::xrange_adaptor<xt::placeholders::xtuph, xt::placeholders::xtuph, long int>, xt::xrange_adaptor<long int, long int, long int>, xt::xrange_adaptor<xt::placeholders::xtuph, xt::placeholders::xtuph, xt::placeholders::xtuph>, xt::xrange<long int>, xt::xstepped_range<long int>, xt::xall_tag, xt::xellipsis_tag, xt::xnewaxis_tag> >]’
/usr/local/include/xtensor/xio.hpp:260:31:   required from ‘void xt::detail::recurser_run(F&, const E&, xt::xstrided_slice_vector&, std::size_t) [with F = xt::detail::printer<xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::svector<long unsigned int, 4, std::allocator<long unsigned int>, true>, xt::xtensor_expression_tag>, void>; E = xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::svector<long unsigned int, 4, std::allocator<long unsigned int>, true>, xt::xtensor_expression_tag>; xt::xstrided_slice_vector = std::vector<mpark::variant<long int, xt::xrange_adaptor<xt::placeholders::xtuph, long int, long int>, xt::xrange_adaptor<long int, xt::placeholders::xtuph, long int>, xt::xrange_adaptor<long int, long int, xt::placeholders::xtuph>, xt::xrange_adaptor<long int, xt::placeholders::xtuph, xt::placeholders::xtuph>, xt::xrange_adaptor<xt::placeholders::xtuph, long int, xt::placeholders::xtuph>, xt::xrange_adaptor<xt::placeholders::xtuph, xt::placeholders::xtuph, long int>, xt::xrange_adaptor<long int, long int, long int>, xt::xrange_adaptor<xt::placeholders::xtuph, xt::placeholders::xtuph, xt::placeholders::xtuph>, xt::xrange<long int>, xt::xstepped_range<long int>, xt::xall_tag, xt::xellipsis_tag, xt::xnewaxis_tag> >; std::size_t = long unsigned int]’
/usr/local/include/xtensor/xio.hpp:763:21:   required from ‘std::ostream& xt::pretty_print(const xt::xexpression<D>&, std::ostream&) [with E = xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::svector<long unsigned int, 4, std::allocator<long unsigned int>, true>, xt::xtensor_expression_tag>; std::ostream = std::basic_ostream<char>]’
/usr/local/include/xtensor/xio.hpp:776:20:   required from ‘std::ostream& xt::operator<<(std::ostream&, const xt::xexpression<D>&) [with E = xt::xarray_container<xt::uvector<double, std::allocator<double> >, (xt::layout_type)1, xt::svector<long unsigned int, 4, std::allocator<long unsigned int>, true>, xt::xtensor_expression_tag>; std::ostream = std::basic_ostream<char>]’
xtensor_example.cu:18:15:   required from here
/usr/local/include/xtensor/xstrided_view_base.hpp:897:6: error: ‘void info’ has incomplete type
                         auto info = xtl::visit(slice_getter, slices[i]);

It seems that a relatively small number of changes could make this work. While I do know some stuff about templates, my knowledge isn't good enough to identify the origin of these errors. I imagine there is some C++17 feature being used here that nvcc doesn't recognize. Since xtensor is supposed to work with C++14, and nvcc is supposed to work with C++14 as well, I thought perhaps this would merit raising an issue.
Any help would be MASSIVELY appreciated.

Hi,
There is no C++17 feature here, only valid C++14, so this looks like a compiler issue ( which does not mean that we should not try to find a workaround). Which version of nvcc do you use?

Thanks Mr. Mabille. Suppose I’ll cook up an MWE and take the nvidia forums then.

We had some surprises with the void_ttrick on other compilers in the past, so I wouldn't be surprised that we are hitting some corner case here. Can you try the following example to see if you still have the first issue?
#include <vector>
#include <array>

#include "xtl/xsquence.hpp"

int main(int argc, char* argv[])
{
    using array_type = std::array<double, 3>;
    using vector_type = std::vector<double>;

    vector_type vsrc = { 0., 1., 2. };
    auto adst = xtl::forward_sequence<array_type>(vsrc);
    std::cout << adst[0] << std::endl;

    array_type asrc = {0., 1., 2. };
    auto vdst = xtl::forward_sequence<vector_type>(asrc);
    std::cout << vdst[0] << std::endl;

    return 0;
}

Hm. This seems to give an error on each of the forward_sequences you declared there. Here are the errors for each:
gavin@gpad:~/scratch$ nvcc mabille_mwe.cu
mabille_mwe.cu(14): error: cannot deduce the return type of function "xtl::forward_sequence<R,A>(A &&) [with R=std::array<double, 3UL>, A=std::vector<double, std::allocator<double>> &]"
/usr/local/include/xtl/xsequence.hpp(33): here; it has not been defined

mabille_mwe.cu(18): error: cannot deduce the return type of function "xtl::forward_sequence<R,A>(A &&) [with R=std::vector<double, std::allocator<double>>, A=std::array<double, 3UL> &]"
/usr/local/include/xtl/xsequence.hpp(33): here; it has not been defined

mabille_mwe.cu(21): error: cannot deduce the return type of function "xtl::forward_sequence<R,A>(A &&) [with R=std::array<double, 3UL>, A=std::array<double, 3UL> &]"
/usr/local/include/xtl/xsequence.hpp(33): here; it has not been defined

mabille_mwe.cu(22): error: cannot deduce the return type of function "xtl::forward_sequence<R,A>(A &&) [with R=std::vector<double, std::allocator<double>>, A=std::vector<double, std::allocator<double>> &]"
/usr/local/include/xtl/xsequence.hpp(33): here; it has not been defined

4 errors detected in the compilation of "/tmp/tmpxft_00000c37_00000000-8_mabille_mwe.cpp1.ii".

I'm not familiar enough with XTL to tell what auto should deduce here, unfortunately.

So, I can confirm that the void_t trick works on this compiler. Wrote this:
#include <iostream>
#include <type_traits>
using namespace std;

// check for a member called asdf
template <typename, typename = void>
struct has_asdf : std::false_type {};
template <typename T>
struct has_asdf<T, std::void_t<decltype(T::asdf)>> : std::true_type {};

// two structs that will be checked for a member called asdf:
struct Duck
{
  double asdf;
};
struct Chicken
{
  double not_asdf;
};

int main()
{
  bool duck_has_asdf = has_asdf<Duck>::value;
  bool chicken_has_asdf = has_asdf<Chicken>::value;
  cout << "duck: " << duck_has_asdf << endl;
  cout << "chicken: " << chicken_has_asdf << endl;
}

And get the expected result from both compilers:
gavin@gpad:~/scratch$ g++ void_t_trick.C
gavin@gpad:~/scratch$ ./a.out
duck: 1
chicken: 0
gavin@gpad:~/scratch$ cp void_t_trick.C void_t_trick.cu
gavin@gpad:~/scratch$ nvcc void_t_trick.cu
gavin@gpad:~/scratch$ ./a.out
duck: 1
chicken: 0

So... what exactly would cause this mechanism to fail, I wonder?

It might be the combination with other things (declval and invoking methods for instance). Thanks to your test, we know that inner type detection works. Out of curiosity, can you test with xtl::void_t instead of std::void_t? I'm pretty sure it will work, but let's be systematic.
Let's fully specify the return type to see what you get:
#include <vector>
#include <array>

#include "xtl/xsquence.hpp"

int main(int argc, char* argv[])
{
    using array_type = std::array<double, 3>;
    using vector_type = std::vector<double>;

    vector_type vsrc = { 0., 1., 2. };
    array_type adst = xtl::forward_sequence<array_type>(vsrc);
    std::cout << adst[0] << std::endl;

    array_type asrc = {0., 1., 2. };
    vector_type vdst = xtl::forward_sequence<vector_type>(asrc);
    std::cout << vdst[0] << std::endl;

    adst = xtl::forward_sequence<array_type>(asrc);
    vdst = xtl::forward_sequence<vector_type>(vsrc);
    std::cout << adst[0] << std::endl;
    std::cout << vdst[0] << std::endl;

    return 0;
}

Good point on checking void_t. So, yes, I can confirm that xmeta_utils.hpp and using xt::void_t does the trick as well.
OK, so, making the change you suggest, I still get the same errors as above regarding "it has not been defined". I suppose it wasn't the auto there but rather something deeper. I'm going to try my best to isolate this, and I'm happy to test any more things if you have ideas. Thanks again for the tips.

Ok, let's try some simplification, can you comment lines 33 and 34 in xsequence.hpp and try again? These are just a forward declaration for readability.

So, yes, it does seem that this made a difference in the error message that's more helpful. Instead, the error becomes
mabille_mwe.cu(14): error: no instance of overloaded function "xtl::forward_sequence" matches the argument list
            argument types are: (vector_type)

mabille_mwe.cu(18): error: no instance of overloaded function "xtl::forward_sequence" matches the argument list
            argument types are: (array_type)

mabille_mwe.cu(21): error: no instance of overloaded function "xtl::forward_sequence" matches the argument list
            argument types are: (array_type)

mabille_mwe.cu(22): error: no instance of overloaded function "xtl::forward_sequence" matches the argument list
            argument types are: (vector_type)

I can't explain why this would happen. My thought for this type of error is that both definitions are being incorrectly SFINAE'd out, but I don't see where this might be happening.
So, something else to consider is that if I try to compile an empty program after including xsequence.hpp, I get this error message:
/usr/local/include/xtl/xsequence.hpp:136:87: error: expected primary-expression before ‘unsigned’
         struct sequence_forwarder_impl<R, A, void_t<decltype(std::declval<R>().resize(std::size_t()))>>

So I suppose figuring this out first would be good. Changing std::size_t() to std::declvalstd::size_t() seems to fix this. Maybe the nvidia compiler isn't correctly grouping the parentheses here or something?

Nice, we are progressing! Good catch for the std::declval<std::size_t>(). The idea was to write something like void_t<decltype(std::declval<R>().resize(0))> but in that case any compiler would see 0 as int and would fail at detecting the existence of resize(size_t). Thus the usage of std::size_t(). I think most of the compilers understand it as the litteral 0. But strictly speaking, we should avoid the constructor and use std::declval instead.
Let's specify all the argument and hope the compiler will give us a more meaningful error message (or at least the list of overloads it considers and why they fail):
#include <vector>
#include <array>

#include "xtl/xsquence.hpp"

int main(int argc, char* argv[])
{
    using array_type = std::array<double, 3>;
    using vector_type = std::vector<double>;

    vector_type vsrc = { 0., 1., 2. };
    array_type adst = xtl::forward_sequence<array_type, vector_type&>(vsrc);
    std::cout << adst[0] << std::endl;

    array_type asrc = {0., 1., 2. };
    vector_type vdst = xtl::forward_sequence<vector_type, array_type&>(asrc);
    std::cout << vdst[0] << std::endl;

    adst = xtl::forward_sequence<array_type, array_type&>(asrc);
    vdst = xtl::forward_sequence<vector_type, vector_type&>(vsrc);
    std::cout << adst[0] << std::endl;
    std::cout << vdst[0] << std::endl;

    return 0;
}

Well, fully specifying the template seems to correctly compile! I suppose the remove_reference stuff is messing up the overloaded functions in some way. I’ll be playing with this...

Ah indeed, the remove_reference is here to prevent the compiler from deducing the type and to force the caller to specify all the arguments. It is tje same principle as the one used in the implementation of std::forward. I should have read the code mire carefully, sorry for missing that!
Would you like to open a PR to fix the two issues you found ? (commenting out the forward declaration and the use of std::declval)

Yes, I’d love to make a PR. However, I’d also like to figure out the trouble with xvariant_impl which appears at the beginning of this issue before doing so, if that’s OK.
I can confirm that the changes we just found fix the error in forward_sequence when trying to compile a basic xtensor program with CUDA though. Yay!

By the way, it seems that leaving the forward declaration in is fine. So it’s up to you whether to take it out.

So, it seems that the maintainer of the variant repository hasn't responded to a similar PR for a bit over a month, so could we maybe go ahead and merge this? The test system in master for variant is failing according to that issue/ still unmerged PR.

Congrats, your patch landed now in upstream: https://github.com/mpark/variant/commit/3c7fc8266bb46046b42c2dc2663f9f505f0cec28 :)

OK, so, when would be good to make the change from mpark/variant to here?

Wow I had totally forgotten this, sorry! I can do it next week, or you can open a PR before if you prefer. We just need to check whether xtensor and xframe build with this change to know if the release will be backward-compatible or not.

No worries! Well, I know that xtensor works with this change since I've run it locally many times. Just thought it would be good to get this change made.

Good to know! So as you wish, if you open the PR I will merge it and release right after. Otherwise, I will do it on Monday.

Fixed with #188


New feature suggestion: choicelist and checkboxes

Inspired from Yeoman and https://medium.com/@jdxcode/12-factor-cli-apps-dd3c227a0e46 (point 7)
Modern CLIs often propose a dynamic choicelist when they can rather than a traditional prompt. Could we imagine integrating such a feature in picocli?
(Off-topic: picocli is great, thanks for having made it!)

Interesting! I wonder if this functionality should live in picocli-shell-jline2: jline combined with picocli gives a good base for interactive CLI features.

I didn't know JLine but it indeed seems to work well as this will require to monitor the arrow keys.

Nice! oclif seems to have tons of interesting features. Very inspiring!
Do you think you'll be able to provide a pull request for any of these?

I think it unlikely, alas, as I probably won't have the time I'd like to look into these. If I succeed however, I'll be sure to fork and share!

There was a nice attempt to build inquirer.js like prompting based on jline: https://github.com/awegmann/consoleui (video)
It seems to be abandoned now, but still could be a good starting point for a new implementation.

Nice find, @xvik!

That sounds like it would give a lot of bang for its buck! Please let me know when you have some good examples.

The one thing I'm noticing about this library is that its not very customizable in terms of the ansi styling. I think it might be fairly straight forward to port, but I just don't have the time at the moment.
All said, I would be willing to add some links and examples to the README.md if that makes sense. Let me know.


Supporting creating a new document virtually

Extensions should be able to create a new panel/document and open it in the editor, much like how markdown preview does.

@cfjedimaster The idea is that you can register for a URI-scheme, like foo and that someone will ask you provide the contents for foo://something/anotherthing when a corresponding call to openTextDocument is made.
Those document must not be able to be saved and can be readonly by some mechanism (TBD). Will that be enough for your needs?

Sure.

Idea is to allow to contribute content (?) provider, like so
interface TextDocumentContentProvider {
  open(uri:Uri):Thenable<string|Buffer>;
  close(uri:Uri):Thenable<any>;
}

registerTextDocumentProvider(scheme | pattern, provider);

Open would require a uri? Does that mean I'd need to save the text locally first?

no, it's more to allow to define your own space using a scheme, like so myextension://project1/path/to/fake/resource would be a valid Uri and it would be up to you to resolve that somehow, like deriving info from a dll-file.

Ok... I kinda get it. :) I'd love to test this when possible. Let me know.

I will keep you in the loop once we sketch out the API for this.

I would love this feature as well :)

Optimistically scheduling this for Jan ;-)

I'd love to see this feature happen!  Since the Markdown preview is being used as an example, does this mean we'd be able to return HTML as content and have it be rendered correctly for rich output?
cc @dfinke

Unsure what you mean by output, but a goal is to allow others to generate markdown and have it shown/previewed in VS Code

Sorry, the term 'output' was a little vague :)  To rephrase, I'm asking if this would allow a content provider to return arbitrary (within reason) HTML to be rendered.  For example, if I create a mode for editing TODO list files with a special text format, it'd be nice if I could register a content provider that can render some nicely-formatted overview details for the user's TODO items.
Returning Markdown to be rendered could also work but would be less flexible.

Hmm, I'd keep it "agnostic". Make the "view" a HTML view so I can send
whatever I want, but making it parse Markdown means I'd have to escape if I
didn't want the automatic conversion.
Basically - make it a web view and I can send whatever I want.
On Tue, Jan 5, 2016 at 9:25 AM, Johannes Rieken<EMAIL_ADDRESS>wrote:

Unsure what you mean by output, but a goal is to allow others to generate
markdown and have it shown/previewed in VS Code
—
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/vscode/issues/272#issuecomment-169033971.

--
Raymond Camden, Developer Advocate for MobileFirst at IBM
Email<EMAIL_ADDRESS>Blog : www.raymondcamden.com
Twitter: raymondcamden

One possibility could be someone editing a CSV file and the VS Code extension would take it as input, render it through d3.js creating a visual and displaying the resulting HTML in the view

Great ideas. I will spin it around with the team. We do have a Markdown_Editor_ and a HTML_Editor_ so it shouldn't be too hard.

It allows things like ionide-webview https://twitter.com/IonideProject/status/648945352100421632

Ionide-webview is very naive - since Atom allows direct DOM access, it's just HTML iframe showing arbitrary webpage.

Being able to pass the view HTML to render would be great, as I'd like to make a extension similar to the Markdown one, which would render ReStructuredText in the preview mode just like parkdown does.

One potential requirement for this feature is that the content provider should be able to get notifications to refresh itself.  @jrieken's proposed API doesn't include that yet, but is it possible that this could be enabled as well?

@daviwil that or the other way around where the provider emits an event to tell the editor to refresh. That would cover the case of virtual.md being generated from on_disk.file.

I might have misunderstood the intent of this feature.  After re-reading your initial comments, it seems like this is to allow an extension to provide a new text editor buffer with dynamically-generated, read-only content, sort of like how Visual Studio can give you a C# buffer with class shapes from an assembly generated from its metadata.
I think I was looking at as a rendered document buffer which appears to be a different thing entirely.  It seems that @cfjedimaster's initial request was along these lines as well if I'm understanding it correctly.  I suppose we'd need a different API for that?

Basically, I want to create a panel of content. For my purposes, the content would be something I'd then copy and paste into a blog. So for example, take my source code of  and convert it to <html>.
The actual "panel" could be read only or not - it doesn't really matter to me. If it was a basic web view then it should support form fields which in theory makes it writeable, but it wouldn't be an "editor" though.
Does that make sense?

Btw, if you get Brackets and my HTMLEscape extension, you can see an example of this in action. I don't expect it to work exactly like Brackets of course. :)

@cfjedimaster Ok, the HTMLEscape extension should be possible with it. The flow could be like so:
(1) register content provider for html-escape-scheme
(2) register command that
(2.1) takes html from an (active) editor
(2.2) generates escape plain text &stores it along an uri with the afore scheme, like html-escape://this/and/that,
(2.3) tells vs code to open that document workbench.openTextDocument(Uri.parse('html-escape://this/and/that'))

"stores it along an uri" - does this mean I need to take the string and escape it for a URL before doing so? Would I have the limit that URLs normally have in browsers (I forget the exact length).

It's just that you are responsible for relating the uri to the string content. You don't need to encode it in the uri (for which btw no length contains apply locally), but you could for instance encode an array index, or the uri of the document to generate from etc. Since the uri travels between the main thread and the extension host short is obviously better. Tho, we might derive a UI label from it (TBD).

Ok. Don't think I grok that 100% quite yet, but when available, I can test. :) As an aside, I'm also happy to test this on a dev build of Code, just let me know. When I was a Brackets user, I used the heck out of my extension as I blog quite a bit.

In theory you can use it off the PR but I'll ping you once it's in master.

Thanks!
On Wed, Jan 6, 2016 at 10:14 AM, Johannes Rieken<EMAIL_ADDRESS>wrote:

In theory you can use it off the PR but I'll ping you once it's in master.
—
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/vscode/issues/272#issuecomment-169373515.

--
Raymond Camden, Developer Advocate for MobileFirst at IBM
Email<EMAIL_ADDRESS>Blog : www.raymondcamden.com
Twitter: raymondcamden

so potentially I can reuse the same to  have an extension which provide a UML view or any other graphical view of my workspace ?

The mechanics to create a virtual document are in, the preview/render html command is tracked in #1946

Feature request: API support for making a virtual document preview-only.
This would allow extensions an easy way to provide a mostly-seamless GUI using an HTML preview and web sockets. (Allowing javascript calls directly between the HTML preview and extension might also be nice...)

@siegebell If you mean 'readonly' by preview then you are ok

@jrieken no, I mean preview-only: there is a "preview" button at the top-right of a document that allows the user to toggle between preview or code view. I am requesting that this button and its keybinding optionally be disabled so that a [virtual] document is stuck in preview mode.
Use case
I am developing a vscode extension for the Coq Proof Assistant, which allows the user to manually step through each line of their proof scipt and see how their hypotheses and proof obligations change. Such interaction falls outside the debug and compilation metaphor that vscode supports in its interface, so I need to write a custom interface to display this information.
Ideally, vscode would provide me an API to write my own interface extensions. Until it does, I've found a nearly sufficient alternative: display an HTML document in preview mode that interacts with my extension using web sockets to display information as the user steps through their proof. It is even interactive. Virtual read-only documents will make this easier.
But if "toggle preview" is still enabled, the user may accidentally switch to the HTML code and become confused. Since viewing the underlying code is of no use to the user, I am requesting that the API provide me a way of:

Turing preview on (versus toggling, which sometimes has strange behavior if the document is already loaded in preview mode from the last session before my extension starts up)
Preventing the user from toggling preview for virtual documents that my extension has created.

no worries, with the new html preview part there is no toggle botton but it will show the path of the URI. having said that, the preview mechanism isn't designed to be a UI extension API. While you can render html and run scripts you should not go crazy in there. The updating (when a document emits a change event) is built for high frequency of changes

I believe that this is now implemented in the Insider build. By any chance is there a sample extension that demonstrates this?

@jrieken suggest to add a sample in https://github.com/Microsoft/vscode-extension-samples

@cfjedimaster As long as we don't have a sample you should check with our api-test: https://github.com/Microsoft/vscode/blob/master/extensions/vscode-api-tests/src/commands.test.ts#L42

Hmm, I appreciate the link, but it doesn't make much sense to me yet how I
would use it. I'm willing to wait though till it is documented more. :)
On Fri, Feb 5, 2016 at 3:01 AM, Johannes Rieken<EMAIL_ADDRESS>wrote:

@cfjedimaster https://github.com/cfjedimaster As long as we don't have
a sample you should check with our api-test:
https://github.com/Microsoft/vscode/blob/master/extensions/vscode-api-tests/src/commands.test.ts#L42
—
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/vscode/issues/272#issuecomment-180258436.

--
Raymond Camden, Developer Advocate for MobileFirst at IBM
Email<EMAIL_ADDRESS>Blog : www.raymondcamden.com
Twitter: raymondcamden

I've added a sample that illustrates how to use TextContentProviders and the vscode.previewHtml commands.
Pls see: https://github.com/Microsoft/vscode-extension-samples/blob/master/textdocumentprovider-sample/README.md

Thank you - will give this a shot!
On Tue, Feb 9, 2016 at 4:30 PM, Erich Gamma<EMAIL_ADDRESS>wrote:

I've added a sample that illustrates how to use TextContentProviders and
the vscode.previewHtml commands.
Pls see:
https://github.com/Microsoft/vscode-extension-samples/blob/master/textdocumentprovider-sample/README.md
—
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/vscode/issues/272#issuecomment-182108020.

--
Raymond Camden, Developer Advocate for MobileFirst at IBM
Email<EMAIL_ADDRESS>Blog : www.raymondcamden.com
Twitter: raymondcamden

Should this work in the most recent mainline release? I'm getting errors
when I try it:
Activating extension eg2.vscode-css-properties failed: Cannot find module
'/Users/raymondcamden/Downloads/vscode-extension-samples-master/textdocumentprovider-sample/out/src/extension'.
Activating extension eg2.vscode-css-properties failed:  Cannot find
module
'/Users/raymondcamden/Downloads/vscode-extension-samples-master/textdocumentprovider-sample/out/src/extension'
On Wed, Feb 10, 2016 at 7:17 AM, Raymond Camden<EMAIL_ADDRESS>wrote:

Thank you - will give this a shot!
On Tue, Feb 9, 2016 at 4:30 PM, Erich Gamma<EMAIL_ADDRESS>wrote:

I've added a sample that illustrates how to use TextContentProviders and
the vscode.previewHtml commands.
Pls see:
https://github.com/Microsoft/vscode-extension-samples/blob/master/textdocumentprovider-sample/README.md
—
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/vscode/issues/272#issuecomment-182108020.

--
Raymond Camden, Developer Advocate for MobileFirst at IBM
Email<EMAIL_ADDRESS>Blog : www.raymondcamden.com
Twitter: raymondcamden

--
Raymond Camden, Developer Advocate for MobileFirst at IBM
Email<EMAIL_ADDRESS>Blog : www.raymondcamden.com
Twitter: raymondcamden

Forgot to run npm install - sorry about that.
On Wed, Feb 10, 2016 at 7:49 AM, Raymond Camden<EMAIL_ADDRESS>wrote:

Should this work in the most recent mainline release? I'm getting errors
when I try it:

--
Raymond Camden, Developer Advocate for MobileFirst at IBM
Email<EMAIL_ADDRESS>Blog : www.raymondcamden.com
Twitter: raymondcamden

I've also published the sample to the market place just in case.

WOOT! I have it working. I'm kinda guessing at a few things and it is 99%
your code still, but it is there. I literally just need to polish it up a
bit and then I can publish mine too. :) Thank you so much.
Quick question. This line:
var previewUri =

vscode.Uri.parse('css-preview://authority/css-preview');
How would you describe that URI? This is what I went with but I was just
guessing:
let previewUri =

vscode.Uri.parse('html-escape://cfjedimaster/html-escape');
On Wed, Feb 10, 2016 at 8:04 AM, Erich Gamma<EMAIL_ADDRESS>wrote:

I've also published the sample to the market place just in case.
—
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/vscode/issues/272#issuecomment-182388477.

--
Raymond Camden, Developer Advocate for MobileFirst at IBM
Email<EMAIL_ADDRESS>Blog : www.raymondcamden.com
Twitter: raymondcamden

And here it is: https://marketplace.visualstudio.com/items?itemName=raymondcamden.htmlescape-vscode-extension

Cool that it worked for you.

vscode.Uri.parse('html-escape://cfjedimaster/html-escape');

This looks good to me, you define your own URI scheme html-escape and all the rest is yours.

So 100% of the string is - basically - whatever we want? Well, within reason, and named something at least similar to what it is doing?

Yes, as long as it properly identifies the resource and conforms to an URI.

@cfjedimaster I ran yo code and updated the package devDependencies for ^0.11.x, also added a "postinstall": "node ./node_modules/vscode/bin/install". Ran npm install.
My extension stills says here is no exported member TextDocumentContentProvider.
I'm missing a step somewhere

Emphasized the missing step to update the engines field in the release notes.


support multi-nodes and multi-gpus lightgbm training base on torch. distributed

Summary

Motivation

Description

References

@CVPaul can you please edit the description and add more details explaining this feature request?

@CVPaul can you please edit the description and add more details explaining this feature request?


docs: Add repo.lock troubleshooting steps to readme

repo.lock issues are frequent, and we don't have an easy place for users to find a solution for this. This PR should help.

@hacdias lidel should be back on Monday, but it would be nice to close some PRs on this repo if you get a chance to peek at this tiny PR.

Thanks @hacdias ! I used all your suggestions =)


Pretty Print Display Logical Plan

@GanZiheng give it a try !

@GanZiheng give it a try !

https://github.com/Veeupup/naive-query-engine/pull/49
@Veeupup PTAL, is this pretty enough?

Amazing work, let me cc : D

Physical Plan, too.


fix: whitespace and argument typo

...because I saw your showcase on Discord 😉
(I guess those were on accident?)

Oops, good catches. Thank you ❤

Happy to fix stuff. 😊 ❤


Cartesian product is in the docs, but does not exist anymore

See https://polyadic.github.io/funcky/book/enumerable-extensions/enumerable-extensions.html#cartesianproduct
Should probably be removed from the docs, or the alternative marked :)

Good catch :)
It should be easy to update the docs with an example using SelectMany (the trick is to pass a result selector in addition to the regular selector).


[kong 2.1.0][DB-less] Prometheus counter not initialied while logging request

Summary
Open the prometheus plugin in DB-less mode to report an error
Steps To Reproduce
1.configure kong
 sed -i "/^#database/i\database = off" /etc/kong/kong.conf

2.configure routes&services&plugins
cat > /tmp/kong.yml <<EOF
_format_version: "1.1"
services:
- name: service-httpbin
    url: http://localhost:8080
    routes:
  - name: v1
    paths:
    - /v1
plugins:
- name: prometheus
EOF

3.start kong
4.update declarative config file
curl -F<EMAIL_ADDRESS>localhost:8001/config

5.open a new terminal and send request
 curl http://<IP_ADDRESS>:8000/v1

6.after a while, view logs/error.log

Additional Details & Logs

Kong version 2.1.0
CentOS Linux release 7.6.1810 (Core)
Hardware 8Core16G

ping @hbagdi

Can you share all the logs of Kong?
This has been previously seen and the root cause here is an error that happens during startup and results in plugins not being initialized correctly. Looking at logs, we can figure out why the initialization fails.

sure
[root@localhost logs]# cat error.log
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"

Seems like your logs were reset. The necessary logs are truncated in the logs you have shared. Can you restart kong and then share the logs?

Steps To Reproduce
1.restart kong
[root@localhost ~]# kong restart
Kong stopped
Kong started

2.update declarative config file
[root@localhost ~]# curl -F<EMAIL_ADDRESS>localhost:8001/config
{"services":{"02721903-3b5a-5c99-a76a-524639a48522":{"host":"localhost","id":"02721903-3b5a-5c99-a76a-524639a48522","protocol":"http","read_timeout":60000,"port":8080,"updated_at":1595981184,"created_at":1595981184,"connect_timeout":60000,"write_timeout":60000,"name":"service-httpbin","retries":5}},"routes":{"7f7e3420-3266-517c-a283-63430f203bbf":{"id":"7f7e3420-3266-517c-a283-63430f203bbf","path_handling":"v0","paths":["\/v1"],"protocols":["http","https"],"created_at":1595981184,"name":"v1","service":{"id":"02721903-3b5a-5c99-a76a-524639a48522"},"preserve_host":false,"regex_priority":0,"https_redirect_status_code":426,"updated_at":1595981184,"strip_path":true}},"plugins":{"86e6f637-64e2-5672-920c-32e17e76e0fe":{"created_at":1595981184,"id":"86e6f637-64e2-5672-920c-32e17e76e0fe","name":"prometheus","enabled":true,"config":{},"protocols":["grpc","grpcs","http","https"]}}}

3.open a new terminal and send request
curl http://<IP_ADDRESS>:8000/v1

Logs
[root@localhost ~]# cat /usr/local/kong/logs/access.log
{"time_local":"29/Jul/2020:07:39:45 +0800","remote_addr":"<IP_ADDRESS>","http_x_forwarded_for":"","remote_user":"","bytes_sent":321,"status":502,"vhost": "<IP_ADDRESS>","request":"GET /v1 HTTP/1.1","request_time":0.001,"request_length":80,"body_bytes_sent":75,"http_referer":"","http_user_agent":"curl/7.70.0","upstream_addr":"<IP_ADDRESS>:8080, <IP_ADDRESS>:8080, <IP_ADDRESS>:8080, <IP_ADDRESS>:8080, <IP_ADDRESS>:8080, <IP_ADDRESS>:8080","upstream_status":"502, 502, 502, 502, 502, 502","upstream_connect_time":"-, -, -, -, -, -","upstream_header_time":"-, -, -, -, -, -","upstream_response_time":"0.000, 0.000, 0.000, 0.000, 0.000, 0.000"}
{"time_local":"29/Jul/2020:07:40:19 +0800","remote_addr":"<IP_ADDRESS>","http_x_forwarded_for":"","remote_user":"","bytes_sent":248,"status":200,"vhost": "<IP_ADDRESS>","request":"GET /v1 HTTP/1.1","request_time":0.004,"request_length":80,"body_bytes_sent":2,"http_referer":"","http_user_agent":"curl/7.70.0","upstream_addr":"<IP_ADDRESS>:8080","upstream_status":"200","upstream_connect_time":"0.000","upstream_header_time":"0.003","upstream_response_time":"0.004"}
{"time_local":"29/Jul/2020:08:09:03 +0800","remote_addr":"<IP_ADDRESS>","http_x_forwarded_for":"","remote_user":"","bytes_sent":248,"status":200,"vhost": "<IP_ADDRESS>","request":"GET /v1 HTTP/1.1","request_time":0.005,"request_length":80,"body_bytes_sent":2,"http_referer":"","http_user_agent":"curl/7.70.0","upstream_addr":"<IP_ADDRESS>:8080","upstream_status":"200","upstream_connect_time":"0.001","upstream_header_time":"0.004","upstream_response_time":"0.004"}

[root@localhost ~]# cat /usr/local/kong/logs/error.log
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 07:40:19 [error] 18867#0: *4478 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:03:50 [notice] 18859#0: signal 15 (SIGTERM) received from 22001, exiting
2020/07/29 08:03:50 [notice] 18860#0: exiting
2020/07/29 08:03:50 [notice] 18861#0: exiting
2020/07/29 08:03:50 [notice] 18862#0: exiting
2020/07/29 08:03:50 [notice] 18866#0: exiting
2020/07/29 08:03:50 [notice] 18865#0: exiting
2020/07/29 08:03:50 [notice] 18867#0: exiting
2020/07/29 08:03:50 [notice] 18863#0: exiting
2020/07/29 08:03:50 [notice] 18864#0: exiting
2020/07/29 08:03:50 [notice] 18861#0: exit
2020/07/29 08:03:50 [notice] 18866#0: exit
2020/07/29 08:03:50 [notice] 18863#0: exit
2020/07/29 08:03:50 [notice] 18867#0: exit
2020/07/29 08:03:50 [notice] 18864#0: exit
2020/07/29 08:03:50 [notice] 18862#0: exit
2020/07/29 08:03:50 [notice] 18859#0: signal 17 (SIGCHLD) received from 18861
2020/07/29 08:03:50 [notice] 18859#0: worker process 18861 exited with code 0
2020/07/29 08:03:50 [notice] 18859#0: worker process 18867 exited with code 0
2020/07/29 08:03:50 [notice] 18859#0: signal 29 (SIGIO) received
2020/07/29 08:03:50 [notice] 18859#0: signal 17 (SIGCHLD) received from 18867
2020/07/29 08:03:50 [notice] 18859#0: signal 17 (SIGCHLD) received from 18866
2020/07/29 08:03:50 [notice] 18859#0: worker process 18862 exited with code 0
2020/07/29 08:03:50 [notice] 18859#0: worker process 18863 exited with code 0
2020/07/29 08:03:50 [notice] 18859#0: worker process 18864 exited with code 0
2020/07/29 08:03:50 [notice] 18859#0: worker process 18866 exited with code 0
2020/07/29 08:03:50 [notice] 18859#0: signal 29 (SIGIO) received
2020/07/29 08:03:50 [notice] 18865#0: exit
2020/07/29 08:03:50 [notice] 18860#0: exit
2020/07/29 08:03:50 [notice] 18859#0: signal 17 (SIGCHLD) received from 18865
2020/07/29 08:03:50 [notice] 18859#0: worker process 18865 exited with code 0
2020/07/29 08:03:50 [notice] 18859#0: signal 29 (SIGIO) received
2020/07/29 08:03:50 [notice] 18859#0: signal 17 (SIGCHLD) received from 18860
2020/07/29 08:03:50 [notice] 18859#0: worker process 18860 exited with code 0
2020/07/29 08:03:50 [notice] 18859#0: exit
2020/07/29 08:03:50 [warn] 22017#0: load balancing method redefined in /usr/local/kong/nginx-kong.conf:60
2020/07/29 08:03:50 [warn] 22024#0: load balancing method redefined in /usr/local/kong/nginx-kong.conf:60
2020/07/29 08:03:50 [notice] 22024#0: using the "epoll" event method
2020/07/29 08:03:50 [notice] 22024#0: openresty/<IP_ADDRESS>
2020/07/29 08:03:50 [notice] 22024#0: built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)
2020/07/29 08:03:50 [notice] 22024#0: OS: Linux 3.10.0-957.5.1.el7.x86_64
2020/07/29 08:03:50 [notice] 22024#0: getrlimit(RLIMIT_NOFILE): 1000000:1000000
2020/07/29 08:03:50 [notice] 22032#0: start worker processes
2020/07/29 08:03:50 [notice] 22032#0: start worker process 22033
2020/07/29 08:03:50 [notice] 22032#0: start worker process 22034
2020/07/29 08:03:50 [notice] 22032#0: start worker process 22035
2020/07/29 08:03:50 [notice] 22032#0: start worker process 22036
2020/07/29 08:03:50 [notice] 22032#0: start worker process 22037
2020/07/29 08:03:50 [notice] 22032#0: start worker process 22038
2020/07/29 08:03:50 [notice] 22032#0: start worker process 22039
2020/07/29 08:03:50 [notice] 22032#0: start worker process 22040
2020/07/29 08:03:50 [notice] 22037#0: *5 [lua] cache.lua:374: purge(): [DB cache] purging (local) cache, context: init_worker_by_lua*
2020/07/29 08:03:50 [notice] 22037#0: *5 [lua] cache.lua:374: purge(): [DB cache] purging (local) cache, context: init_worker_by_lua*
2020/07/29 08:03:50 [warn] 22036#0: *3 [lua] globalpatches.lua:55: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
2020/07/29 08:03:50 [warn] 22034#0: *2 [lua] globalpatches.lua:55: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
2020/07/29 08:05:29 [notice] 22039#0: *899 [lua] cache.lua:374: purge(): [DB cache] purging (local) cache, client: <IP_ADDRESS>, server: kong_admin, request: "POST /config HTTP/1.1", host: "localhost:8001"
2020/07/29 08:05:29 [notice] 22039#0: *899 [lua] cache.lua:374: purge(): [DB cache] purging (local) cache, client: <IP_ADDRESS>, server: kong_admin, request: "POST /config HTTP/1.1", host: "localhost:8001"
2020/07/29 08:06:24 [notice] 22037#0: *1406 [lua] cache.lua:374: purge(): [DB cache] purging (local) cache, client: <IP_ADDRESS>, server: kong_admin, request: "POST /config HTTP/1.1", host: "localhost:8001"
2020/07/29 08:06:24 [notice] 22037#0: *1406 [lua] cache.lua:374: purge(): [DB cache] purging (local) cache, client: <IP_ADDRESS>, server: kong_admin, request: "POST /config HTTP/1.1", host: "localhost:8001"
2020/07/29 08:08:52 [error] 22039#0: *2750 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:08:52 [error] 22039#0: *2750 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:08:52 [error] 22039#0: *2750 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:08:52 [error] 22039#0: *2750 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:08:52 [error] 22039#0: *2750 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:08:52 [error] 22039#0: *2750 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:09:03 [error] 22038#0: *2856 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:09:03 [error] 22038#0: *2856 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:09:03 [error] 22038#0: *2856 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:09:03 [error] 22038#0: *2856 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:09:03 [error] 22038#0: *2856 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"
2020/07/29 08:09:03 [error] 22038#0: *2856 [lua] prometheus.lua:671: log_error(): counter not initialied while logging request, client: <IP_ADDRESS>, server: kong, request: "GET /v1 HTTP/1.1", upstream: "http://<IP_ADDRESS>:8080/", host: "<IP_ADDRESS>:8000"

ping @hbagdi

ping @fffonion

@cciechao I didn't reproduce it locally, which distribution for 2.1.0 you are using? Is it source code, docker or binary package?

Did not use binary installation method to install, currently tried rpm installation and docker operation, and both have the same problem
[root@localhost ~]# kong version -vv
2020/07/29 13:18:22 [verbose] Kong: 2.1.0
2020/07/29 13:18:22 [debug] ngx_lua: 10015
2020/07/29 13:18:22 [debug] nginx: 1015008
2020/07/29 13:18:22 [debug] Lua: LuaJIT 2.1.0-beta3
2.1.0

@fffonion

cool, could you share the docker environment variables, args passed to kong container and kong.conf if there's any?

There is a binary installation machine, the adjusted configuration content is as follows
[root@localhost ~]# grep -vE '#|^$' /etc/kong/kong.conf
admin_listen = <IP_ADDRESS>:8001 reuseport backlog=16384, <IP_ADDRESS>:8444 http2 ssl reuseport backlog=16384
database = off

@fffonion

Now it rings a bell to me, I think that's because the init_worker phase are not ran with config reload. @cciechao The temporary fix is to fed kong with an initial declarative config that includes the prometheus plugin.
I will update the issue title.

Maybe related to #6011, which seems not in 2.1.0.

I ran some tests here and it seems that the init_worker phases are being run for plugins on config reload.
Is it possible that errors in creating new counters from lua-resty-counter are not being caught? The new function is not returning the expected nil, err [1] [2].
I was able to get those counter not initialied log entries by forcing errors in the new method in lua-resty-counter.

@locao I can see one condition for this is to have more than one workers, since the init_worker phase looks like only runs on worker 0
2020/08/10 17:32:57 [debug] 22#0: *1 [lua] counter.lua:70: new(): start timer for shdict kong on worker 0
2020/08/10 17:32:57 [debug] 23#0: *2 [lua] counter.lua:70: new(): start timer for shdict kong on worker 1
2020/08/10 17:32:57 [info] 22#0: *1 [kong] handler.lua:53 [acme] acme renew timer started on worker 0, context: init_worker_by_lua*
2020/08/10 17:32:57 [debug] 25#0: *3 [lua] counter.lua:70: new(): start timer for shdict kong on worker 3
2020/08/10 17:32:57 [debug] 22#0: *1 [lua] counter.lua:70: new(): start timer for shdict prometheus_metrics on worker 0
2020/08/10 17:32:57 [debug] 24#0: *4 [lua] counter.lua:70: new(): start timer for shdict kong on worker 2

That was a great investigation, thanks @fffonion! It seems that the plugins_iterator was being wrongfully built, but it wasn't the same issue fixed by #6011. I opened a PR that should fix this.


clinfo crashes computer ubuntu 20.04

When running clinfo it crashes the computer, or at the very least visually crashes video.
This is on a 3 agent system.
Threadripper 1950x
RX 580
RX 580

You could always try a fresh install WITH recommended distribution AND recommended kernel. You can find those in documentation.

Thank you for reaching out. You can find AMD ROCm documentation at:
http://rocmdocs.amd.com
If you are unable to find the instructions you need, please let us know.
AMD ROCm Documentation Team

This repository is for developers correct?
I am merely letting developers know of a bug on a platform I expect will be supported in the future.
Everything works fine when you have 1 GPU, but if you have 2 it crashes video.

Thank you for letting the community know. You are right, Ubuntu 20.04 is not supported as yet.

Not sure if this is the same case, but clinfo will crash after installing ROCm. By crash I mean it will cause a 'Segmentation fault (core dump)'. I have to install libtinfo5, or libncurses5 before it will work. Anyway, after some trial and error I have rocm-dev working on my computer with Ubuntu 20.04. Well, the OpenCL part anyway, I haven't tried other parts.

Thanks for letting me know.  Pretty sure I had both of these installed when attempting to get it going, but I can give it a go again.  When I attempted to run things other then clinfo it totally locked up video.  Tensorflow did not want to work at all.

Version 3.5 for Ubuntu 20.04 change nothing. Same errors like for 3.3...

The AMD ROCm v3.5.x platform is designed to support the following operating systems:

Ubuntu 16.04.6(Kernel 4.15) and 18.04.4(Kernel 5.3)
CentOS 7.7 (Kernel 3.10-1062) and RHEL 7.8(Kernel 3.10.0-1127)(Using devtoolset-7 runtime support)
SLES 15 SP1
CentOS and RHEL 8.1(Kernel 4.18.0-147)

Yes, I hope 4.0 release will fix Ubuntu 20.04...

Hi @Goddard
Thanks for reaching us.
I am closing this issue as the latest ROCm 4.0 supports Ubuntu 20.04.x and hope this problem is gone with ROCm 4.0+Ubuntu 20.04.x.
Request you to try the same.
Request you to open a new issue if any, so that we can chase down the problem.
Thank you.

Thanks @Goddard and @alfabuster
We are officially supporting the latest ROCm 20.04 and request you to try the same now.
Ubuntu 20.04 + ROCm 4.0 works perfect and I recommend you to wait you wish to try the latest ROCm 4.1, which is mostly a week away.
Thank you.


Functional test

Four functional tests have been added in this PR (a big step on #7) :

preprocessing
training
inference on default validation dataset
inference on unknown dataset

The reference dataset for these functional tests is a small subset of S3DIS.
The target is not to build relevant models through functional tests. This PR aims at verifying that the main processes are working fine by producing relevant material. To this end, a bunch of parameters have been added to the CLI, in order to manage the training and inference process durations.
Here comes the new parameters :

training case:

max_epoch : number of training epochs
checkpoint_gap : checkpoint saving frequency (in terms of epochs)
epoch_steps : number of training steps in each epoch
validation_size : number of validation steps after each training epoch

inference case:

num_votes : threshold that min potentials mush reach during inference (solves #10)
potential_increment : prediction saving frequency (in terms of min potential increment)
validation_size : number of steps in each inference iteration (min potential is computed after each iteration)

So as to make these functional tests OK, a bunch of small improvements have been done on the code :

Migrate some occurrence of os.path to pathlib.Path.
Change the content of running_PID.txtfile (written during training process, and removed at the end of it).
Remove running_PID.txt at the end of the last epoch instead of removing it at the beginning, and shift the epoch value on some saved material.
When using an existing checkpoint, append new training metrics to the existing training.txt file instead of overload it.
Do not kill (by os.kill) the training module run at the end of the execution, just let the run finish smoothly.

Bonus: pytest-dependency added as a new development dependency, so as to indicates that the inference functional tests depend on the training inference test.
Related issues : #7, #10

The min potential increment is not strictly equal on the tested Python version, hence the current result saving strategy is buggy.
I will add a commit for modifying it: instead of saving inference result when min_potential is roughly a multiple of X, I will propose to save inference results if min_potentials increases of X since last save.

Suggestion: add a pytest.ini file
[pytest]
markers =
    dependency: tests depending one from another

to avoid warnings.

Suggestion: add a pytest.ini file
[pytest]
markers =
    dependency: tests depending one from another

to avoid warnings.

@AuBousquet which warning do you have?

Suggestion: add a pytest.ini file
[pytest]
markers =
    dependency: tests depending one from another

to avoid warnings.

@AuBousquet which warning do you have?

@delhomer
tests/test_train.py:49
  /home/abo/kpconv-torch/tests/test_train.py:49: PytestUnknownMarkWarning: Unknown pytest.mark.dependency - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.dependency()

Suggestion: add a pytest.ini file
[pytest]
markers =
    dependency: tests depending one from another

to avoid warnings.

@AuBousquet which warning do you have?

@delhomer
tests/test_train.py:49
  /home/abo/kpconv-torch/tests/test_train.py:49: PytestUnknownMarkWarning: Unknown pytest.mark.dependency - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @mark.dependency()

Ok, I think you need pip install pytest-dependency (see the change in requirements-dev.txt).

@AuBousquet could you please have a look on that PR? Big diff because of the fixture files, the real useful changes are drastically smaller (I promise :-) ).

Ce serait bien d'ajouter les paramètres de CLI à la doc, dans le fichier docs/cli.md.

Ce serait bien d'ajouter les paramètres de CLI à la doc, dans le fichier docs/cli.md.

Done. I will merge as soon as the CI becomes green again.


fix: #739 - MockDirectory.GetDirectories empty searchOption

This PR fixes #739 and also adds corresponding tests.

Thanks a  lot!


feat: Block cheat codes in anvil

Adds a second api to the mainnet-fork, located at
public-PUBLIC_API_KEY, in which all cheat codes are blocked. We define a cheat code as a method in the evm, hardhat, or anvil namespaces. We use njs for parsing the JSON RPC request body and testing the method namespace.
The PUBLIC_API_KEY is only set for provernet, for other networks, we load the same API_KEY secret as always.
This PR also enables logging for anvil, including a logrotate config to ensure we don't fill up the disk just with logs.

Benchmark results
Metrics with a significant change:

protocol_circuit_simulation_time_in_ms (base-rollup): 3,445 (+17%)
avm_simulation_time_ms (Token:assert_minter_and_mint): 66.5 (-79%)
avm_simulation_time_ms (Token:_increase_public_balance): 39.0 (-33%)
avm_simulation_bytecode_size_in_bytes (FeeJuice:set_portal): 2,128 (-48%)
avm_simulation_bytecode_size_in_bytes (FeeJuice:mint_public): 5,415 (-17%)
avm_simulation_bytecode_size_in_bytes (Token:mint_public): 10,101 (-20%)
avm_simulation_bytecode_size_in_bytes (Token:assert_minter_and_mint): 6,844 (-19%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Proof generation
Each column represents the number of threads used in proof generation.

Metric
1 threads
4 threads
16 threads
32 threads
64 threads

proof_construction_time_sha256_ms
5,763
1,557
713
765 (+1%)
772 (-1%)

proof_construction_time_sha256_30_ms
11,470 (-1%)
3,090 (-1%)
1,388 (-2%)
1,425 (-2%)
1,473 (-1%)

proof_construction_time_sha256_100_ms
44,022
11,788
5,407
5,575 (-4%)
5,501 (-3%)

proof_construction_time_poseidon_hash_ms
78.0
34.0 (-28%)
34.0
58.0 (+2%)
88.0 (+2%)

proof_construction_time_poseidon_hash_30_ms
1,529
421
203
225
271

proof_construction_time_poseidon_hash_100_ms
5,647
1,508 (-1%)
674
720 (-3%)
748

L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
4 txs
8 txs
16 txs

l1_rollup_calldata_size_in_bytes
4,356
7,876
14,884

l1_rollup_calldata_gas
50,244
92,980
178,012

l1_rollup_execution_gas
842,904
1,576,536
3,361,624

l2_block_processing_time_in_ms
258 (+2%)
451 (+1%)
832 (+1%)

l2_block_building_time_in_ms
11,412
22,247
44,535

l2_block_rollup_simulation_time_in_ms
11,412
22,247
44,535

l2_block_public_tx_process_time_in_ms
9,649
20,424 (-1%)
42,663

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 8 txs.

Metric
3 blocks
5 blocks

node_history_sync_time_in_ms
3,007 (+2%)
4,012 (+5%)

node_database_size_in_bytes
12,595,280
16,711,760

pxe_database_size_in_bytes
16,254
26,813

Circuits stats
Stats on running time and I/O sizes collected for every kernel circuit run across all benchmarks.

Circuit
simulation_time_in_ms
witness_generation_time_in_ms
input_size_in_bytes
output_size_in_bytes
proving_time_in_ms

private-kernel-init
95.8
398 (+1%)
21,735
44,860
N/A

private-kernel-inner
173 (+1%)
704
72,544
45,007
N/A

private-kernel-reset-tiny
307
712 (-4%)
65,593
44,846
N/A

private-kernel-tail
167
135 (-1%)
50,644
52,257
N/A

base-parity
5.61 (+1%)
N/A
160
96.0
N/A

root-parity
35.5 (-1%)
N/A
73,948
96.0
N/A

base-rollup
:warning: 3,445 (+17%)
N/A
189,136
664
N/A

block-root-rollup
41.4
N/A
58,205
2,448
N/A

public-kernel-setup
84.4
N/A
105,085
71,222
N/A

public-kernel-app-logic
97.5
N/A
104,911
71,222
N/A

public-kernel-tail
857
N/A
390,582
16,414
N/A

private-kernel-reset-small
308
N/A
66,341
45,629
N/A

private-kernel-tail-to-public
1,374
625 (+1%)
455,400
1,825
N/A

public-kernel-teardown
83.8 (-1%)
N/A
105,349
71,222
N/A

merge-rollup
20.7 (+3%)
N/A
38,174
664
N/A

undefined
N/A
N/A
N/A
N/A
79,258 (+1%)

Stats on running time collected for app circuits

Function
input_size_in_bytes
output_size_in_bytes
witness_generation_time_in_ms

ContractClassRegisterer:register
1,344
11,731
343 (+1%)

ContractInstanceDeployer:deploy
1,408
11,731
18.3 (+1%)

MultiCallEntrypoint:entrypoint
1,920
11,731
404

FeeJuice:deploy
1,376
11,731
386

SchnorrAccount:constructor
1,312
11,731
73.9 (+1%)

SchnorrAccount:entrypoint
2,304 (-1%)
11,731
395 (+1%)

Token:privately_mint_private_note
1,280
11,731
107 (+5%)

FPC:fee_entrypoint_public
1,344
11,731
26.8 (+5%)

Token:transfer
1,312
11,731
224 (-3%)

Benchmarking:create_note
1,344
11,731
86.7 (+2%)

SchnorrAccount:verify_private_authwit
1,280
11,731
27.7 (+1%)

Token:unshield
1,376
11,731
515 (-1%)

FPC:fee_entrypoint_private
1,376
11,731
734 (+7%)

AVM Simulation
Time to simulate various public functions in the AVM.

Function
time_ms
bytecode_size_in_bytes

FeeJuice:_increase_public_balance
60.5 (+6%)
7,101 (-13%)

FeeJuice:set_portal
12.6 (+15%)
:warning: 2,128 (-48%)

Token:constructor
80.7
25,285 (-13%)

FPC:constructor
54.3
17,853 (-6%)

FeeJuice:mint_public
39.2 (-4%)
:warning: 5,415 (-17%)

Token:mint_public
43.6 (-8%)
:warning: 10,101 (-20%)

Token:assert_minter_and_mint
:warning: 66.5 (-79%)
:warning: 6,844 (-19%)

AuthRegistry:set_authorized
37.2 (-24%)
3,969 (-5%)

FPC:prepare_fee
232 (-3%)
6,747

Token:transfer_public
22.0 (-26%)
38,126 (-4%)

FPC:pay_refund
51.1 (-19%)
9,398

Benchmarking:increment_balance
1,199 (-1%)
6,179 (-15%)

Token:_increase_public_balance
:warning: 39.0 (-33%)
7,705 (-11%)

FPC:pay_refund_with_shielded_rebate
62.5 (+1%)
9,881

Public DB Access
Time to access various public DBs.

Function
time_ms

get-nullifier-index
0.163 (+3%)

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
256 leaves
512 leaves
1024 leaves

batch_insert_into_append_only_tree_16_depth_ms
2.21
3.93
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.8
31.7
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.115 (+1%)
0.111
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
11.3 (+1%)
17.7 (+1%)
31.2 (+2%)
59.7 (+1%)
114 (-1%)

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
95.9
159
287
543
1,055

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.109 (+1%)
0.103 (+1%)
0.101 (+1%)
0.103 (+2%)
0.100 (-2%)

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
14.4
26.0 (+1%)
43.8 (+1%)
82.9
161 (-2%)

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
109
207
355
691
1,363

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.109 (-1%)
0.105
0.106 (+1%)
0.102
0.102 (-2%)

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
16.6
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
132
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
0.106
N/A
N/A
N/A
N/A

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes
1 registered classes

tx_size_in_bytes
64,779
668,997

Transaction size based on fee payment method
| Metric |  |
| - |  |


Sankey Chart in Power BI

I have been trying to modify the Sankey Chart made by @davidebacci:
https://github.com/PBI-David/Deneb-Showcase/blob/main/Sankey Chart/Spec.json
The template he has shared is beautiful. However, all of the values have been manually placed into the visualization template.
How can I modify the code to use my Power BI dataset :

Attached is my dataset

Please can you close and ask on Stack Overflow. The Sankey was updated this week to show how to take values from the dataset so check you have the latest file with two report pages in. Also see: https://stackoverflow.com/questions/78891258/integrating-powerbi-measures-into-deneb-vega-visualisation-david-deneb-sankey

Thank you!


Sometimes prompt Model Builder Error when training Data classification scenario.

System Information (please complete the following information):

Microsoft Visual Studio Enterprise 2019: 16.11.5
ML.Net Model Builder: <IP_ADDRESS>1403 (Main Branch)

Describe the bug

On which step of the process did you run into an issue: training Data classification scenario
Clear description of the problem: This issue doesn't occur 100%, it maybe completed without error when you train it again.

To Reproduce
Steps to reproduce the behavior:

Select Create a new project from the Visual Studio 2019 start window;
Choose the C# Console App (.NET Core) project template with .Net 5.0;
Add model builder by right click on the project;
Select Data Classification>Local (CPU)>File (.csv, .tsv, .txt) and input the data source;
Navigate to Train page, click "Start training" button;
Prompt Model Builder Error when the training is about be completed.

Expected behavior
No error during training.
Screenshots
If applicable, add screenshots to help explain your problem.

Error Log
Sentiment_MLModel1-W9LV8R.txt

Need epic for Object null errors.

This shouldn't be happening anymore per XiaoYun's change https://github.com/dotnet/machinelearning-modelbuilder/issues/1513


Make an escape for fast focus

I'm all for escape valves when needed, but can you document the situation that currently necessitates one?

It can cause weird situations on mobile. When I press a text field in iOS (dont know about android and windows) the keyboard will appear and the scroll position of the page might change. If the scroll position changes as such that a link (#link-to) will be on the same position as where the text field was pressed, the link will pressed too. And that causes a transition to the route of the link. One solution is to change the position of the link, another is to disable fast focus.

Will re-open if docs are provided and use case still relevant


[Suggestion] Weapons made of modded materials

If this is within your grasp, perhaps you could try and add some compat to such mods, and give their materials weapons here.

yeah id love compatibility with mods like better nether and better end, great mod btw

dev has given up. you'll need to find another mod.


Advanced Content

I've never seen a workshop at a hackathon go over more advanced features, I think it's worth adding information on things like:

git reflog
git hooks (both on your own computer and on GitHub)

running unit tests every time you try to make a commit (and how to bypass them)
adding integrations to your repository (like Travis-CI)

git alias

And some other web interface content like:

Adding a contributing.md
Pull request and issue templates

Adding this to the Pro Git section to be featured after overview of GitHub features and tutorial.


Finer granularity error codes in order to support i18nized errors from the server

Right now there are many APIs where we return a generic 400,403,404 etc with a human readable error in English but no finer-granularity error code which a client could pick up and present a proper i18nized error message.  Surely we need some kind of solution for this - whether it's either mandating servers do the i18n for errors themselves based on Accept language headers, or having much more sophisticated and finer granularity errors.
A good example is the groups API:
Matthews-MacBook-Pro:synapse matthew$ grep -ri 'group' . | fgrep \" | grep -i error | fgrep .py:
./groups/groups_server.py:            raise SynapseError(400, "Group not on this server")
./groups/groups_server.py:            raise SynapseError(404, "Unknown group")
./groups/groups_server.py:            raise SynapseError(404, "Unknown group")
./groups/groups_server.py:                raise SynapseError(403, "User is not admin in group")
./groups/groups_server.py:            raise SynapseError(404, "Unknown group")
./groups/groups_server.py:            raise SynapseError(403, "User not in group")
./groups/groups_server.py:            raise SynapseError(403, "User not invited to group")
./groups/groups_server.py:            raise SynapseError(403, "Group is not publicly joinable")
./groups/groups_server.py:                raise SynapseError(403, "User is not admin in group")
./groups/groups_server.py:            raise SynapseError(400, "Group already exists")
./groups/attestations.py:            raise SynapseError(400, "Attestation has incorrect group_id")
./groups/attestations.py:            raise SynapseError(400, "Neither user not group are on this server")
./storage/group_server.py:            raise SynapseError(400, "room not in group")
./storage/group_server.py:            raise SynapseError(400, "user not in group")
./federation/transport/server.py:            raise SynapseError(403, "group_id doesn't match origin")

Where clients will not only want to i18nize but also s/group/communities/ for presentation to the user (see https://github.com/vector-im/riot-web/issues/6003).
Related: https://github.com/matrix-org/matrix-doc/issues/1638 and https://github.com/matrix-org/matrix-doc/issues/603

More granular error codes are spread out over quite a few issues in the repo ftr.

You already started doing it with M_ errors - please just do it across the board. Just adding l10n to the server is unlikely to be much beneficial as there's more than one homeserver implementation anyway; I'd say that it's a good work for client SDKs.


Fix Citizens repository url

That's not the URL (that's the name)
That change does nothing (no difference in functionality)
That change is wrong (it's supposed to be what I have it as, what you changed it to is not anything reasonable)
You have no description on this PR nor pre-discussion of the change made (why did you make this PR???)


cmake: Update gaps in config checks

I did some sed-foo to try to figure out the check defines that were used in auto tools, but missing from cmake.  This PR is most of them.

Total of 299 test
  Unknown: 0
  Exception: 0
  Failed: 9
  Warning: 0
  Skipped: 17
  Passed: 273

TSVConnFd
basic_plugin_handler
certifier
redirect_actions
tls_0rtt_server
tls_client_verify3
tls_client_versions
tls_ticket
verify_global_plugin


Word.spellcheck() method returns inconsistent results

I'm currently playing around with TextBlob and I found that spellcheck() method is giving me a lot of false positives. For example, for the word 'upload', the spellcheck() output is the following: [('unload', 1.0)] where I would expect something more similar to the following: [('upload', 0.64), ('unload'), 0.36]. Another example is with the plurals. For instance, given the word "building" the spellcheck() output is the following: [('buildings', 1.0)]. Is this the expected behavior? Is there something I'm missing?

Yes this is the expected behaviour. The words you are providing are not in the frequency dictionary so it is correcting them to next possible suggestion. I think textblob needs to add the better and faster spellcorrect algo like symspell for its corrector. Then these problems will be resolved.

TextBlob just delegates to pattern.en for spellchecking. Feel free to send a PR with an improved spell checker.


regex != pattern

Throughout the readme, comments and code itself the term 'regex' is used.
This is misleading, lua patterns aren't regular expressions so you probably shouldn't hint to the users that they are.
They should probably be renamed to 'pattern' where appropriate

Replaced "regex" with "pattern" where possible (see commit b3746338c69c8fee1ae45ae9a3984a6b1258f79f). string.*regex*() methods are deprecated. But as far as I know, strictly saying, Lua patterns are regular expressions too


Fix the icon rotation issue

A different (and better) fix for #834 as opposed to the way originally proposed. Manipulate the vertex scaling and rotation in "pixel" values before converting back to "fraction" in the screen space.  No need to calculate the aspect ratio and factor it in the rotation function. To do this, it is better to pass the anchor position into the shader as pixel values instead of normalizing it (see the change in icon-layer.js). The devicePixelRatio is used to avoid hardcoding in the previous implementation.

Can you review this @Pessimistress ?

@1chandu @Pessimistress Changed the default texture filtering method to improve texture rendering quality as suggested by @ibgreen

@rivulet-zhang Did it make a difference? Any before/after screen shots?

@ibgreen The quality did improve, especially when the icon is pretty small. See comparison below:
Before:

After:

The quality did improve, especially when the icon is pretty small

@rivulet-zhang that looks like a huge improvements to my eyes.
@tangential

This PR fixed issue #834 #895