AngularJS Custom Directive replace HTML

The code works with element.replaceWith but is this the correct way to replace HTML using custom directive ?

Nope. It is more or less the worst possible wrong way, because it prevents any Angular data or event binding or really any of the other reasons Angular is worth using; you're dumping a blob of uncompiled HTML into the page that Angular doesn't really know about.

You never want to modify the DOM directly in Angular. If you do modify the DOM directly, what you add should almost always be $compiled first so Angular can find any data or event bindings inside it. (Experienced Angular devs may take issue with my use of the word "never". It's true that there are some cases where this advice doesn't apply, but as a new-to-Angular developer you're unlikely to run into them for quite a while. Best practice is to let Angular do the work until you have a good understanding of what it's doing. And then still let Angular do most of the work, because that's what it's good at.)

For your example, the way you've done it, the ng-hide wont work, and if the value of scope.cameraRTSPUrl changes, that change will not be reflected in the UI. Instead, use a directive template or templateUrl, so you'll get access to Angular's data binding:

myApp.directive('videodata', function() {
    return {
        restrict: 'EA',
        scope : true,
        replace: true,
        templateUrl: 'templates/foo.html',
        link: function(scope, element, attrs) {
            scope.cameraRTSPUrl = "someGeneratedValue.mov"; // could passed into the directive via an attribute or parent data binding, for example
        }
    };
});

foo.html:

<div ng-hide="StartPlay" id="streamingarea">
    <object events="True" id="vlc" codebase="http://downloads.videolan.org/pub/videolan/vlc/latest/win32/axvlc.cab" 
        classid="clsid:9BE31822-FDAD-461B-AD51-BE1D1C159921">
        <embed width="800" height="600" target="{{cameraRTSPUrl}}" loop="no" autoplay="yes" 
        version="VideoLAN.VLCPlugin.2" type="application/x-google-vlc-plugin" id="123">
    </object>   
</div>

Now you can trigger the ng-hide by just changing the truthiness of scope.StartPlay, and changes to scope.cameraRTSPUrl will appear in the DOM automatically.

How do I use transclude: true, property in order to replace my HTML?

That's not really what transclude is for. Transclude allows you to create directives that contain html not generated from within that directive (the part that comes from elsewhere is the "transcluded" bit; the transclude directive is a wrapper around that transcluded html.)

Best way is to use template

template: 'Name: {{customer.name}} Address: {{customer.address}}'

 myApp.directive('videodata', function() {
    return {
        restrict: 'EA',
        scope : true ,
        link: function(scope, element, attrs) {
        template: '<object classid="clsid:9BE31822-FDAD-461B-AD51-BE1D1C159921" '+
                                    'codebase="http://downloads.videolan.org/pub/videolan/vlc/latest/win32/axvlc.cab" '+
                                    'id="vlc" events="True"> '+
                                    '<embed id="123" type="application/x-google-vlc-plugin" version="VideoLAN.VLCPlugin.2" autoplay="yes" '+
                                    'loop="no" width="800" height="600" target="'+scope.cameraRTSPUrl+'"> '+
                                '</object>'
        }
    };
});